Restructure solution layout by module
Some checks failed
Docs CI / lint-and-preview (push) Has been cancelled
Some checks failed
Docs CI / lint-and-preview (push) Has been cancelled
This commit is contained in:
root
File diff suppressed because it is too large
Load Diff
@@ -45,7 +45,7 @@ runtime wiring, CLI usage) and leaves connector/internal customization for later
|
||||
4. Start the web service from the repository root:
|
||||
|
||||
```bash
|
||||
dotnet run --project src/StellaOps.Concelier.WebService
|
||||
dotnet run --project src/Concelier/StellaOps.Concelier.WebService
|
||||
```
|
||||
|
||||
On startup Concelier validates the options, boots MongoDB indexes, loads plug-ins,
|
||||
@@ -94,7 +94,7 @@ Rollout checkpoints for the two Authority toggles:
|
||||
## 2 · Configure the CLI
|
||||
|
||||
The CLI reads configuration from JSON/YAML files *and* environment variables. The
|
||||
defaults live in `src/StellaOps.Cli/appsettings.json` and expect overrides at runtime.
|
||||
defaults live in `src/Cli/StellaOps.Cli/appsettings.json` and expect overrides at runtime.
|
||||
|
||||
| Setting | Environment variable | Default | Purpose |
|
||||
| ------- | -------------------- | ------- | ------- |
|
||||
@@ -123,12 +123,12 @@ export STELLAOPS_AUTHORITY_URL="https://authority.local"
|
||||
export STELLAOPS_AUTHORITY_CLIENT_ID="concelier-cli"
|
||||
export STELLAOPS_AUTHORITY_CLIENT_SECRET="s3cr3t"
|
||||
export STELLAOPS_AUTHORITY_SCOPE="concelier.jobs.trigger advisory:ingest advisory:read"
|
||||
dotnet run --project src/StellaOps.Cli -- db merge
|
||||
dotnet run --project src/Cli/StellaOps.Cli -- db merge
|
||||
|
||||
# Acquire a bearer token and confirm cache state
|
||||
dotnet run --project src/StellaOps.Cli -- auth login
|
||||
dotnet run --project src/StellaOps.Cli -- auth status
|
||||
dotnet run --project src/StellaOps.Cli -- auth whoami
|
||||
dotnet run --project src/Cli/StellaOps.Cli -- auth login
|
||||
dotnet run --project src/Cli/StellaOps.Cli -- auth status
|
||||
dotnet run --project src/Cli/StellaOps.Cli -- auth whoami
|
||||
```
|
||||
|
||||
Refer to `docs/dev/32_AUTH_CLIENT_GUIDE.md` for deeper guidance on tuning retry/offline settings and rollout checklists.
|
||||
@@ -143,31 +143,31 @@ rely on environment variables for ephemeral runners.
|
||||
1. **Trigger connector fetch stages**
|
||||
|
||||
```bash
|
||||
dotnet run --project src/StellaOps.Cli -- db fetch --source osv --stage fetch
|
||||
dotnet run --project src/StellaOps.Cli -- db fetch --source osv --stage parse
|
||||
dotnet run --project src/StellaOps.Cli -- db fetch --source osv --stage map
|
||||
dotnet run --project src/Cli/StellaOps.Cli -- db fetch --source osv --stage fetch
|
||||
dotnet run --project src/Cli/StellaOps.Cli -- db fetch --source osv --stage parse
|
||||
dotnet run --project src/Cli/StellaOps.Cli -- db fetch --source osv --stage map
|
||||
```
|
||||
|
||||
Use `--mode resume` when continuing from a previous window:
|
||||
|
||||
```bash
|
||||
dotnet run --project src/StellaOps.Cli -- db fetch --source redhat --stage fetch --mode resume
|
||||
dotnet run --project src/Cli/StellaOps.Cli -- db fetch --source redhat --stage fetch --mode resume
|
||||
```
|
||||
|
||||
2. **Merge canonical advisories**
|
||||
|
||||
```bash
|
||||
dotnet run --project src/StellaOps.Cli -- db merge
|
||||
dotnet run --project src/Cli/StellaOps.Cli -- db merge
|
||||
```
|
||||
|
||||
3. **Produce exports**
|
||||
|
||||
```bash
|
||||
# JSON tree (vuln-list style)
|
||||
dotnet run --project src/StellaOps.Cli -- db export --format json
|
||||
dotnet run --project src/Cli/StellaOps.Cli -- db export --format json
|
||||
|
||||
# Trivy DB (delta example)
|
||||
dotnet run --project src/StellaOps.Cli -- db export --format trivy-db --delta
|
||||
dotnet run --project src/Cli/StellaOps.Cli -- db export --format trivy-db --delta
|
||||
```
|
||||
|
||||
Concelier always produces a deterministic OCI layout. The first run after a clean
|
||||
@@ -207,13 +207,13 @@ rely on environment variables for ephemeral runners.
|
||||
```bash
|
||||
export STELLA_TENANT="${STELLA_TENANT:-tenant-a}"
|
||||
|
||||
dotnet run --project src/StellaOps.Cli -- aoc verify \
|
||||
dotnet run --project src/Cli/StellaOps.Cli -- aoc verify \
|
||||
--since 24h \
|
||||
--format table \
|
||||
--tenant "$STELLA_TENANT"
|
||||
|
||||
# Optional: capture JSON evidence for pipelines/audits
|
||||
dotnet run --project src/StellaOps.Cli -- aoc verify \
|
||||
dotnet run --project src/Cli/StellaOps.Cli -- aoc verify \
|
||||
--since 7d \
|
||||
--limit 100 \
|
||||
--format json \
|
||||
@@ -244,9 +244,9 @@ rely on environment variables for ephemeral runners.
|
||||
6. **Manage scanners (optional)**
|
||||
|
||||
```bash
|
||||
dotnet run --project src/StellaOps.Cli -- scanner download --channel stable
|
||||
dotnet run --project src/StellaOps.Cli -- scan run --entry scanners/latest/Scanner.dll --target ./sboms
|
||||
dotnet run --project src/StellaOps.Cli -- scan upload --file results/scan-001.json
|
||||
dotnet run --project src/Cli/StellaOps.Cli -- scanner download --channel stable
|
||||
dotnet run --project src/Cli/StellaOps.Cli -- scan run --entry scanners/latest/Scanner.dll --target ./sboms
|
||||
dotnet run --project src/Cli/StellaOps.Cli -- scan upload --file results/scan-001.json
|
||||
```
|
||||
|
||||
Add `--verbose` to any command for structured console logs. All commands honour
|
||||
|
||||
@@ -1,380 +1,380 @@
|
||||
# StellaOps Authority Service
|
||||
|
||||
> **Status:** Drafted 2025-10-12 (CORE5B.DOC / DOC1.AUTH) – aligns with Authority revocation store, JWKS rotation, and bootstrap endpoints delivered in Sprint 1.
|
||||
|
||||
## 1. Purpose
|
||||
The **StellaOps Authority** service issues OAuth2/OIDC tokens for every StellaOps module (Concelier, Backend, Agent, Zastava) and exposes the policy controls required in sovereign/offline environments. Authority is built as a minimal ASP.NET host that:
|
||||
|
||||
- brokers password, client-credentials, and device-code flows through pluggable identity providers;
|
||||
- persists access/refresh/device tokens in MongoDB with deterministic schemas for replay analysis and air-gapped audit copies;
|
||||
- distributes revocation bundles and JWKS material so downstream services can enforce lockouts without direct database access;
|
||||
- offers bootstrap APIs for first-run provisioning and key rotation without redeploying binaries.
|
||||
|
||||
Authority is deployed alongside Concelier in air-gapped environments and never requires outbound internet access. All trusted metadata (OpenIddict discovery, JWKS, revocation bundles) is cacheable, signed, and reproducible.
|
||||
|
||||
## 2. Component Architecture
|
||||
Authority is composed of five cooperating subsystems:
|
||||
|
||||
1. **Minimal API host** – configures OpenIddict endpoints (`/token`, `/authorize`, `/revoke`, `/jwks`), publishes the OpenAPI contract at `/.well-known/openapi`, and enables structured logging/telemetry. Rate limiting hooks (`AuthorityRateLimiter`) wrap every request.
|
||||
2. **Plugin host** – loads `StellaOps.Authority.Plugin.*.dll` assemblies, applies capability metadata, and exposes password/client provisioning surfaces through dependency injection.
|
||||
3. **Mongo storage** – persists tokens, revocations, bootstrap invites, and plugin state in deterministic collections indexed for offline sync (`authority_tokens`, `authority_revocations`, etc.).
|
||||
4. **Cryptography layer** – `StellaOps.Cryptography` abstractions manage password hashing, signing keys, JWKS export, and detached JWS generation.
|
||||
5. **Offline ops APIs** – internal endpoints under `/internal/*` provide administrative flows (bootstrap users/clients, revocation export) guarded by API keys and deterministic audit events.
|
||||
|
||||
A high-level sequence for password logins:
|
||||
|
||||
```
|
||||
Client -> /token (password grant)
|
||||
-> Rate limiter & audit hooks
|
||||
-> Plugin credential store (Argon2id verification)
|
||||
-> Token persistence (Mongo authority_tokens)
|
||||
-> Response (access/refresh tokens + deterministic claims)
|
||||
```
|
||||
|
||||
## 3. Token Lifecycle & Persistence
|
||||
Authority persists every issued token in MongoDB so operators can audit or revoke without scanning distributed caches.
|
||||
|
||||
- **Collection:** `authority_tokens`
|
||||
- **Key fields:**
|
||||
- `tokenId`, `type` (`access_token`, `refresh_token`, `device_code`, `authorization_code`)
|
||||
- `subjectId`, `clientId`, ordered `scope` array
|
||||
- `tenant` (lower-cased tenant hint from the issuing client, omitted for global clients)
|
||||
|
||||
### Console OIDC client
|
||||
|
||||
- **Client ID**: `console-web`
|
||||
- **Grants**: `authorization_code` (PKCE required), `refresh_token`
|
||||
- **Audience**: `console`
|
||||
- **Scopes**: `openid`, `profile`, `email`, `advisory:read`, `vex:read`, `aoc:verify`, `findings:read`, `orch:read`, `vuln:read`
|
||||
- **Redirect URIs** (defaults): `https://console.stella-ops.local/oidc/callback`
|
||||
- **Post-logout redirect**: `https://console.stella-ops.local/`
|
||||
- **Tokens**: Access tokens inherit the global 2 minute lifetime; refresh tokens remain short-lived (30 days) and can be exchanged silently via `/token`.
|
||||
- **Roles**: Assign Authority role `Orch.Viewer` (exposed to tenants as `role/orch-viewer`) when operators need read-only access to Orchestrator telemetry via Console dashboards. Policy Studio ships dedicated roles (`role/policy-author`, `role/policy-reviewer`, `role/policy-approver`, `role/policy-operator`, `role/policy-auditor`) that align with the new `policy:*` scope family; issue them per tenant so audit trails remain scoped.
|
||||
|
||||
Configuration sample (`etc/authority.yaml.sample`) seeds the client with a confidential secret so Console can negotiate the code exchange on the backend while browsers execute the PKCE dance.
|
||||
|
||||
### Console Authority endpoints
|
||||
|
||||
- `/console/tenants` — Requires `authority:tenants.read`; returns the tenant catalogue for the authenticated principal. Requests lacking the `X-Stella-Tenant` header are rejected (`tenant_header_missing`) and logged.
|
||||
- `/console/profile` — Requires `ui.read`; exposes subject metadata (roles, scopes, audiences) and indicates whether the session is within the five-minute fresh-auth window.
|
||||
- `/console/token/introspect` — Requires `ui.read`; introspects the active access token so the SPA can prompt for re-authentication before privileged actions.
|
||||
|
||||
All endpoints demand DPoP-bound tokens and propagate structured audit events (`authority.console.*`). Gateways must forward the `X-Stella-Tenant` header derived from the access token; downstream services rely on the same value for isolation. Keep Console access tokens short-lived (default 15 minutes) and enforce the fresh-auth window for admin actions (`ui.admin`, `authority:*`, `policy:activate`, `exceptions:approve`).
|
||||
- `status` (`valid`, `revoked`, `expired`), `createdAt`, optional `expiresAt`
|
||||
- `revokedAt`, machine-readable `revokedReason`, optional `revokedReasonDescription`
|
||||
- `revokedMetadata` (string dictionary for plugin-specific context)
|
||||
- **Persistence flow:** `PersistTokensHandler` stamps missing JWT IDs, normalises scopes, and stores every principal emitted by OpenIddict.
|
||||
- **Revocation flow:** `AuthorityTokenStore.UpdateStatusAsync` flips status, records the reason metadata, and is invoked by token revocation handlers and plugin provisioning events (e.g., disabling a user).
|
||||
- **Expiry maintenance:** `AuthorityTokenStore.DeleteExpiredAsync` prunes non-revoked tokens past their `expiresAt` timestamp. Operators should schedule this in maintenance windows if large volumes of tokens are issued.
|
||||
|
||||
### Expectations for resource servers
|
||||
Resource servers (Concelier WebService, Backend, Agent) **must not** assume in-memory caches are authoritative. They should:
|
||||
|
||||
- cache `/jwks` and `/revocations/export` responses within configured lifetimes;
|
||||
- honour `revokedReason` metadata when shaping audit trails;
|
||||
- treat `status != "valid"` or missing tokens as immediate denial conditions.
|
||||
- propagate the `tenant` claim (`X-Stella-Tenant` header in REST calls) and reject requests when the tenant supplied by Authority does not match the resource server's scope; Concelier and Excititor guard endpoints refuse cross-tenant tokens.
|
||||
|
||||
### Tenant propagation
|
||||
|
||||
- Client provisioning (bootstrap or plug-in) accepts a `tenant` hint. Authority normalises the value (`trim().ToLowerInvariant()`) and persists it alongside the registration. Clients without an explicit tenant remain global.
|
||||
- Issued principals include the `stellaops:tenant` claim. `PersistTokensHandler` mirrors this claim into `authority_tokens.tenant`, enabling per-tenant revocation and reporting.
|
||||
- Rate limiter metadata now tags requests with `authority.tenant`, unlocking per-tenant throughput metrics and diagnostic filters. Audit events (`authority.client_credentials.grant`, `authority.password.grant`, bootstrap flows) surface the tenant and login attempt documents index on `{tenant, occurredAt}` for quick queries.
|
||||
- Client credentials that request `advisory:ingest`, `advisory:read`, `vex:ingest`, `vex:read`, `signals:read`, `signals:write`, `signals:admin`, or `aoc:verify` now fail fast when the client registration lacks a tenant hint. Issued tokens are re-validated against persisted tenant metadata, and Authority rejects any cross-tenant replay (`invalid_client`/`invalid_token`), ensuring aggregation-only workloads remain tenant-scoped.
|
||||
- Client credentials that request `export.viewer`, `export.operator`, or `export.admin` must provide a tenant hint. Requests for `export.admin` also need accompanying `export_reason` and `export_ticket` parameters; Authority returns `invalid_request` when either value is missing and records the denial in token audit events.
|
||||
- Policy Studio scopes (`policy:author`, `policy:review`, `policy:approve`, `policy:operate`, `policy:audit`, `policy:simulate`, `policy:run`, `policy:activate`) require a tenant assignment; Authority rejects tokens missing the hint with `invalid_client` and records `scope.invalid` metadata for auditing.
|
||||
- **AOC pairing guardrails** – Tokens that request `advisory:read`, `vex:read`, or any `signals:*` scope must also request `aoc:verify`. Authority rejects mismatches with `invalid_scope` (`Scope 'aoc:verify' is required when requesting advisory/vex read scopes.` or `Scope 'aoc:verify' is required when requesting signals scopes.`) so automation surfaces deterministic errors.
|
||||
- **Signals ingestion guardrails** – Sensors and services requesting `signals:write`/`signals:admin` must also request `aoc:verify`; Authority records the `authority.aoc_scope_violation` tag when the pairing is missing so operators can trace failing sensors immediately.
|
||||
- Password grant flows reuse the client registration's tenant and enforce the configured scope allow-list. Requested scopes outside that list (or mismatched tenants) trigger `invalid_scope`/`invalid_client` failures, ensuring cross-tenant access is denied before token issuance.
|
||||
|
||||
### Default service scopes
|
||||
|
||||
| Client ID | Purpose | Scopes granted | Sender constraint | Tenant |
|
||||
|----------------------|---------------------------------------|--------------------------------------|-------------------|-----------------|
|
||||
| `concelier-ingest` | Concelier raw advisory ingestion | `advisory:ingest`, `advisory:read` | `dpop` | `tenant-default` |
|
||||
| `excitor-ingest` | Excititor raw VEX ingestion | `vex:ingest`, `vex:read` | `dpop` | `tenant-default` |
|
||||
| `aoc-verifier` | Aggregation-only contract verification | `aoc:verify`, `advisory:read`, `vex:read` | `dpop` | `tenant-default` |
|
||||
| `cartographer-service` | Graph snapshot construction | `graph:write`, `graph:read` | `dpop` | `tenant-default` |
|
||||
| `graph-api` | Graph Explorer gateway/API | `graph:read`, `graph:export`, `graph:simulate` | `dpop` | `tenant-default` |
|
||||
| `export-center-operator` | Export Center operator automation | `export.viewer`, `export.operator` | `dpop` | `tenant-default` |
|
||||
| `export-center-admin` | Export Center administrative automation | `export.viewer`, `export.operator`, `export.admin` | `dpop` | `tenant-default` |
|
||||
| `vuln-explorer-ui` | Vuln Explorer UI/API | `vuln:read` | `dpop` | `tenant-default` |
|
||||
| `signals-uploader` | Reachability sensor ingestion | `signals:write`, `signals:read`, `aoc:verify` | `dpop` | `tenant-default` |
|
||||
|
||||
> **Secret hygiene (2025‑10‑27):** The repository includes a convenience `etc/authority.yaml` for compose/helm smoke tests. Every entry’s `secretFile` points to `etc/secrets/*.secret`, which ship with `*-change-me` placeholders—replace them with strong values (and wire them through your vault/secret manager) before issuing tokens in CI, staging, or production.
|
||||
|
||||
For factory provisioning, issue sensors the **SignalsUploader** role template (`signals:write`, `signals:read`, `aoc:verify`). Authority rejects ingestion tokens that omit `aoc:verify`, preserving aggregation-only contract guarantees for reachability signals.
|
||||
|
||||
These registrations are provided as examples in `etc/authority.yaml.sample`. Clone them per tenant (for example `concelier-tenant-a`, `concelier-tenant-b`) so tokens remain tenant-scoped by construction.
|
||||
|
||||
Graph Explorer introduces dedicated scopes: `graph:write` for Cartographer build jobs, `graph:read` for query/read operations, `graph:export` for long-running export downloads, and `graph:simulate` for what-if overlays. Assign only the scopes a client actually needs to preserve least privilege—UI-facing clients should typically request read/export access, while background services (Cartographer, Scheduler) require write privileges.
|
||||
|
||||
#### Least-privilege guidance for graph clients
|
||||
|
||||
- **Service identities** – The Cartographer worker should request `graph:write` and `graph:read` only; grant `graph:simulate` exclusively to pipeline automation that invokes Policy Engine overlays on demand. Keep `graph:export` scoped to API gateway components responsible for streaming GraphML/JSONL artifacts. Authority enforces this by rejecting `graph:write` tokens that lack `properties.serviceIdentity: cartographer`.
|
||||
- **Tenant propagation** – Every client registration must pin a `tenant` hint. Authority normalises the value and stamps it into issued tokens (`stellaops:tenant`) so downstream services (Scheduler, Graph API, Console) can enforce tenant isolation without custom headers. Graph scopes (`graph:read`, `graph:write`, `graph:export`, `graph:simulate`) are denied if the tenant hint is missing.
|
||||
- **SDK alignment** – Use the generated `StellaOpsScopes` constants in service code to request graph scopes. Hard-coded strings risk falling out of sync as additional graph capabilities are added.
|
||||
- **DPOP for automation** – Maintain sender-constrained (`dpop`) flows for Cartographer and Scheduler to limit reuse of access tokens if a build host is compromised. For UI-facing tokens, pair `graph:read`/`graph:export` with short lifetimes and enforce refresh-token rotation at the gateway.
|
||||
|
||||
#### Export Center scope guardrails
|
||||
|
||||
- **Viewer vs operator** – `export.viewer` grants read-only access to export profiles, manifests, and bundles. Automation that schedules or reruns exports should request `export.operator` (and typically `export.viewer`). Tenant hints remain mandatory; Authority refuses tokens without them.
|
||||
- **Administrative mutations** – Changes to retention policies, encryption key references, or schedule defaults require `export.admin`. When requesting tokens with this scope, clients must supply `export_reason` and `export_ticket` parameters; Authority persists the values for audit records and rejects missing metadata with `invalid_request`.
|
||||
- **Operational hygiene** – Rotate `export.admin` credentials infrequently and run them through fresh-auth workflows where possible. Prefer distributing verification tooling with `export.viewer` tokens for day-to-day bundle validation.
|
||||
|
||||
#### Vuln Explorer permalinks
|
||||
|
||||
- **Scope** – `vuln:read` authorises Vuln Explorer to fetch advisory/linkset evidence and issue shareable links. Assign it only to front-end/API clients that must render vulnerability details.
|
||||
- **Signed links** – `POST /permalinks/vuln` (requires `vuln:read`) accepts `{ "tenant": "tenant-a", "resourceKind": "vulnerability", "state": { ... }, "expiresInSeconds": 86400 }` and returns a JWT (`token`) plus `issuedAt`/`expiresAt`. The token embeds the tenant, requested state, and `vuln:read` scope and is signed with the same Authority signing keys published via `/jwks`.
|
||||
- **Validation** – Resource servers verify the permalink using cached JWKS: check signature, ensure the tenant matches the current request context, honour the expiry, and enforce the contained `vuln:read` scope. The payload’s `resource.state` block is opaque JSON so UIs can round-trip filters/search terms without new schema changes.
|
||||
|
||||
## 4. Revocation Pipeline
|
||||
Authority centralises revocation in `authority_revocations` with deterministic categories:
|
||||
|
||||
| Category | Meaning | Required fields |
|
||||
| --- | --- | --- |
|
||||
| `token` | Specific OAuth token revoked early. | `revocationId` (token id), `tokenType`, optional `clientId`, `subjectId` |
|
||||
| `subject` | All tokens for a subject disabled. | `revocationId` (= subject id) |
|
||||
| `client` | OAuth client registration revoked. | `revocationId` (= client id) |
|
||||
| `key` | Signing/JWE key withdrawn. | `revocationId` (= key id) |
|
||||
|
||||
`RevocationBundleBuilder` flattens Mongo documents into canonical JSON, sorts entries by (`category`, `revocationId`, `revokedAt`), and signs exports using detached JWS (RFC 7797) with cosign-compatible headers.
|
||||
|
||||
**Export surfaces** (deterministic output, suitable for Offline Kit):
|
||||
|
||||
- CLI: `stella auth revoke export --output ./out` writes `revocation-bundle.json`, `.jws`, `.sha256`.
|
||||
- Verification: `stella auth revoke verify --bundle <path> --signature <path> --key <path>` validates detached JWS signatures before distribution, selecting the crypto provider advertised in the detached header (see `docs/security/revocation-bundle.md`).
|
||||
- API: `GET /internal/revocations/export` (requires bootstrap API key) returns the same payload.
|
||||
- Verification: `stella auth revoke verify` validates schema, digest, and detached JWS using cached JWKS or offline keys, automatically preferring the hinted provider (libsodium builds honour `provider=libsodium`; other builds fall back to the managed provider).
|
||||
|
||||
**Consumer guidance:**
|
||||
|
||||
1. Mirror `revocation-bundle.json*` alongside Concelier exports. Offline agents fetch both over the existing update channel.
|
||||
2. Use bundle `sequence` and `bundleId` to detect replay or monotonicity regressions. Ignore bundles with older sequence numbers unless `bundleId` changes and `issuedAt` advances.
|
||||
3. Treat `revokedReason` taxonomy as machine-friendly codes (`compromised`, `rotation`, `policy`, `lifecycle`). Translating to human-readable logs is the consumer’s responsibility.
|
||||
|
||||
## 5. Signing Keys & JWKS Rotation
|
||||
Authority signs revocation bundles and publishes JWKS entries via the new signing manager:
|
||||
|
||||
- **Configuration (`authority.yaml`):**
|
||||
```yaml
|
||||
signing:
|
||||
enabled: true
|
||||
algorithm: ES256 # Defaults to ES256
|
||||
keySource: file # Loader identifier (file, vault, etc.)
|
||||
provider: default # Optional preferred crypto provider
|
||||
activeKeyId: authority-signing-dev
|
||||
keyPath: "../certificates/authority-signing-dev.pem"
|
||||
additionalKeys:
|
||||
- keyId: authority-signing-dev-2024
|
||||
path: "../certificates/authority-signing-dev-2024.pem"
|
||||
source: "file"
|
||||
```
|
||||
- **Sources:** The default loader supports PEM files relative to the content root; additional loaders can be registered via `IAuthoritySigningKeySource`.
|
||||
- **Providers:** Keys are registered against the `ICryptoProviderRegistry`, so alternative implementations (HSM, libsodium) can be plugged in without changing host code.
|
||||
- **OpenAPI discovery:** `GET /.well-known/openapi` returns the published authentication contract (JSON by default, YAML when requested). Responses include `X-StellaOps-Service`, `X-StellaOps-Api-Version`, `X-StellaOps-Build-Version`, plus grant and scope headers, and honour conditional requests via `ETag`/`If-None-Match`.
|
||||
- **JWKS output:** `GET /jwks` lists every signing key with `status` metadata (`active`, `retired`). Old keys remain until operators remove them from configuration, allowing verification of historical bundles/tokens.
|
||||
|
||||
### Rotation SOP (no downtime)
|
||||
1. Generate a new P-256 private key (PEM) on an offline workstation and place it where the Authority host can read it (e.g., `../certificates/authority-signing-2025.pem`).
|
||||
2. Call the authenticated admin API:
|
||||
```bash
|
||||
curl -sS -X POST https://authority.example.com/internal/signing/rotate \
|
||||
-H "x-stellaops-bootstrap-key: ${BOOTSTRAP_KEY}" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"keyId": "authority-signing-2025",
|
||||
"location": "../certificates/authority-signing-2025.pem",
|
||||
"source": "file"
|
||||
}'
|
||||
```
|
||||
3. Verify the response reports the previous key as retired and fetch `/jwks` to confirm the new `kid` appears with `status: "active"`.
|
||||
4. Persist the old key path in `signing.additionalKeys` (the rotation API updates in-memory options; rewrite the YAML to match so restarts remain consistent).
|
||||
5. If you prefer automation, trigger the `.gitea/workflows/authority-key-rotation.yml` workflow with the new `keyId`/`keyPath`; it wraps `ops/authority/key-rotation.sh` and reads environment-specific secrets. The older key will be marked `retired` and appended to `signing.additionalKeys`.
|
||||
6. Re-run `stella auth revoke export` so revocation bundles are signed with the new key. Downstream caches should refresh JWKS within their configured lifetime (`StellaOpsAuthorityOptions.Signing` + client cache tolerance).
|
||||
|
||||
The rotation API leverages the same cryptography abstractions as revocation signing; no restart is required and the previous key is marked `retired` but kept available for verification.
|
||||
|
||||
## 6. Bootstrap & Administrative Endpoints
|
||||
Administrative APIs live under `/internal/*` and require the bootstrap API key plus rate-limiter compliance.
|
||||
|
||||
| Endpoint | Method | Description |
|
||||
| --- | --- | --- |
|
||||
| `/internal/users` | `POST` | Provision initial administrative accounts through the registered password-capable plug-in. Emits structured audit events. |
|
||||
| `/internal/clients` | `POST` | Provision OAuth clients (client credentials / device code). |
|
||||
| `/internal/revocations/export` | `GET` | Export revocation bundle + detached JWS + digest. |
|
||||
| `/internal/signing/rotate` | `POST` | Promote a new signing key (see SOP above). Request body accepts `keyId`, `location`, optional `source`, `algorithm`, `provider`, and metadata. |
|
||||
|
||||
All administrative calls emit `AuthEventRecord` entries enriched with correlation IDs, PII tags, and network metadata for offline SOC ingestion.
|
||||
|
||||
> **Tenant hint:** include a `tenant` entry inside `properties` when bootstrapping clients. Authority normalises the value, stores it on the registration, and stamps future tokens/audit events with the tenant.
|
||||
|
||||
### Bootstrap client example
|
||||
|
||||
```jsonc
|
||||
POST /internal/clients
|
||||
{
|
||||
"clientId": "concelier",
|
||||
"confidential": true,
|
||||
"displayName": "Concelier Backend",
|
||||
"allowedGrantTypes": ["client_credentials"],
|
||||
"allowedScopes": ["concelier.jobs.trigger", "advisory:ingest", "advisory:read"],
|
||||
"properties": {
|
||||
"tenant": "tenant-default"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
For environments with multiple tenants, repeat the call per tenant-specific client (e.g. `concelier-tenant-a`, `concelier-tenant-b`) or append suffixes to the client identifier.
|
||||
|
||||
### Aggregation-only verification tokens
|
||||
|
||||
- Issue a dedicated client (e.g. `aoc-verifier`) with the scopes `aoc:verify`, `advisory:read`, and `vex:read` for each tenant that runs guard checks. Authority refuses to mint tokens for these scopes unless the client registration provides a tenant hint.
|
||||
- The CLI (`stella aoc verify --tenant <tenant>`) and Console verification panel both call `/aoc/verify` on Concelier and Excititor. Tokens that omit the tenant claim or present a tenant that does not match the stored registration are rejected with `invalid_client`/`invalid_token`.
|
||||
- Audit: `authority.client_credentials.grant` entries record `scope.invalid="aoc:verify"` when requests are rejected because the tenant hint is missing or mismatched.
|
||||
|
||||
### Exception approvals & routing
|
||||
|
||||
- New scopes `exceptions:read`, `exceptions:write`, and `exceptions:approve` govern access to the exception lifecycle. Map these via tenant roles (`exceptions-service`, `exceptions-approver`) as described in `/docs/security/authority-scopes.md`.
|
||||
- Configure approval routing in `authority.yaml` with declarative templates. Each template exposes an `authorityRouteId` for downstream services (Policy Engine, Console) and an optional `requireMfa` flag:
|
||||
|
||||
```yaml
|
||||
exceptions:
|
||||
routingTemplates:
|
||||
- id: "secops"
|
||||
authorityRouteId: "approvals/secops"
|
||||
requireMfa: true
|
||||
description: "Security Operations approval chain"
|
||||
- id: "governance"
|
||||
authorityRouteId: "approvals/governance"
|
||||
requireMfa: false
|
||||
description: "Non-production waiver review"
|
||||
```
|
||||
|
||||
- Clients requesting exception scopes must include a tenant assignment. Authority rejects client-credential flows that request `exceptions:*` with `invalid_client` and logs `scope.invalid="exceptions:write"` (or the requested scope) in `authority.client_credentials.grant` audit events when the tenant hint is missing.
|
||||
- When any configured routing template sets `requireMfa: true`, user-facing tokens that contain `exceptions:approve` must be acquired through an MFA-capable identity provider. Password/OIDC flows that lack MFA support are rejected with `authority.password.grant` audit events where `reason="Exception approval scope requires an MFA-capable identity provider."`
|
||||
- Update interactive clients (Console) to request `exceptions:read` by default and elevate to `exceptions:approve` only inside fresh-auth workflows for approvers. Documented examples live in `etc/authority.yaml.sample`.
|
||||
- Verification responses map guard failures to `ERR_AOC_00x` codes and Authority emits `authority.client_credentials.grant` + `authority.token.validate_access` audit records containing the tenant and scopes so operators can trace who executed a run.
|
||||
- For air-gapped or offline replicas, pre-issue verification tokens per tenant and rotate them alongside ingest credentials; the guard endpoints never mutate data and remain safe to expose through the offline kit schedule.
|
||||
|
||||
## 7. Configuration Reference
|
||||
|
||||
| Section | Key | Description | Notes |
|
||||
| --- | --- | --- | --- |
|
||||
| Root | `issuer` | Absolute HTTPS issuer advertised to clients. | Required. Loopback HTTP allowed only for development. |
|
||||
| Tokens | `accessTokenLifetime`, `refreshTokenLifetime`, etc. | Lifetimes for each grant (access, refresh, device, authorization code, identity). | Enforced during issuance; persisted on each token document. |
|
||||
| Storage | `storage.connectionString` | MongoDB connection string. | Required even for tests; offline kits ship snapshots for seeding. |
|
||||
| Signing | `signing.enabled` | Enable JWKS/revocation signing. | Disable only for development. |
|
||||
| Signing | `signing.algorithm` | Signing algorithm identifier. | Currently ES256; additional curves can be wired through crypto providers. |
|
||||
| Signing | `signing.keySource` | Loader identifier (`file`, `vault`, custom). | Determines which `IAuthoritySigningKeySource` resolves keys. |
|
||||
| Signing | `signing.keyPath` | Relative/absolute path understood by the loader. | Stored as-is; rotation request should keep it in sync with filesystem layout. |
|
||||
| Signing | `signing.activeKeyId` | Active JWKS / revocation signing key id. | Exposed as `kid` in JWKS and bundles. |
|
||||
| Signing | `signing.additionalKeys[].keyId` | Retired key identifier retained for verification. | Manager updates this automatically after rotation; keep YAML aligned. |
|
||||
| Signing | `signing.additionalKeys[].source` | Loader identifier per retired key. | Defaults to `signing.keySource` if omitted. |
|
||||
| Security | `security.rateLimiting` | Fixed-window limits for `/token`, `/authorize`, `/internal/*`. | See `docs/security/rate-limits.md` for tuning. |
|
||||
| Bootstrap | `bootstrap.apiKey` | Shared secret required for `/internal/*`. | Only required when `bootstrap.enabled` is true. |
|
||||
|
||||
### 7.1 Sender-constrained clients (DPoP & mTLS)
|
||||
|
||||
Authority now understands two flavours of sender-constrained OAuth clients:
|
||||
|
||||
- **DPoP proof-of-possession** – clients sign a `DPoP` header for `/token` requests. Authority validates the JWK thumbprint, HTTP method/URI, and replay window, then stamps the resulting access token with `cnf.jkt` so downstream services can verify the same key is reused.
|
||||
- Configure under `security.senderConstraints.dpop`. `allowedAlgorithms`, `proofLifetime`, and `replayWindow` are enforced at validation time.
|
||||
- `security.senderConstraints.dpop.nonce.enabled` enables nonce challenges for high-value audiences (`requiredAudiences`, normalised to case-insensitive strings). When a nonce is required but missing or expired, `/token` replies with `WWW-Authenticate: DPoP error="use_dpop_nonce"` (and, when available, a fresh `DPoP-Nonce` header). Clients must retry with the issued nonce embedded in the proof.
|
||||
- `security.senderConstraints.dpop.nonce.store` selects `memory` (default) or `redis`. When `redis` is configured, set `security.senderConstraints.dpop.nonce.redisConnectionString` so replicas share nonce issuance and high-value clients avoid replay gaps during failover.
|
||||
- Example (enabling Redis-backed nonces; adjust audiences per deployment):
|
||||
```yaml
|
||||
security:
|
||||
senderConstraints:
|
||||
dpop:
|
||||
enabled: true
|
||||
proofLifetime: "00:02:00"
|
||||
replayWindow: "00:05:00"
|
||||
allowedAlgorithms: [ "ES256", "ES384" ]
|
||||
nonce:
|
||||
enabled: true
|
||||
ttl: "00:10:00"
|
||||
maxIssuancePerMinute: 120
|
||||
store: "redis"
|
||||
redisConnectionString: "redis://authority-redis:6379?ssl=false"
|
||||
requiredAudiences:
|
||||
- "signer"
|
||||
- "attestor"
|
||||
```
|
||||
Operators can override any field via environment variables (e.g. `STELLAOPS_AUTHORITY__SECURITY__SENDERCONSTRAINTS__DPOP__NONCE__STORE=redis`).
|
||||
- Declare client `audiences` in bootstrap manifests or plug-in provisioning metadata; Authority now defaults the token `aud` claim and `resource` indicator from this list, which is also used to trigger nonce enforcement for audiences such as `signer` and `attestor`.
|
||||
- **Mutual TLS clients** – client registrations may declare an mTLS binding (`senderConstraint: mtls`). When enabled via `security.senderConstraints.mtls`, Authority validates the presented client certificate against stored bindings (`certificateBindings[]`), optional chain verification, and timing windows. Successful requests embed `cnf.x5t#S256` into the access token (and introspection output) so resource servers can enforce the certificate thumbprint.
|
||||
- `security.senderConstraints.mtls.enforceForAudiences` forces mTLS whenever the requested `aud`/`resource` (or the client's configured audiences) intersect the configured allow-list (default includes `signer`). Clients configured for different sender constraints are rejected early so operator policy remains consistent.
|
||||
- Certificate bindings now act as an allow-list: Authority verifies thumbprint, subject, issuer, serial number, and any declared SAN values against the presented certificate, with rotation grace windows applied to `notBefore/notAfter`. Operators can enforce subject regexes, SAN type allow-lists (`dns`, `uri`, `ip`), trusted certificate authorities, and rotation grace via `security.senderConstraints.mtls.*`.
|
||||
|
||||
Both modes persist additional metadata in `authority_tokens`: `senderConstraint` records the enforced policy, while `senderKeyThumbprint` stores the DPoP JWK thumbprint or mTLS certificate hash captured at issuance. Downstream services can rely on these fields (and the corresponding `cnf` claim) when auditing offline copies of the token store.
|
||||
|
||||
### 7.2 Policy Engine clients & scopes
|
||||
|
||||
Policy Engine v2 introduces dedicated scopes and a service identity that materialises effective findings. Configure Authority as follows when provisioning policy clients:
|
||||
|
||||
| Client | Scopes | Notes |
|
||||
| --- | --- | --- |
|
||||
| `policy-engine` (service) | `policy:run`, `findings:read`, `effective:write` | Must include `properties.serviceIdentity: policy-engine` and a tenant. Authority rejects `effective:write` tokens without the marker or tenant. |
|
||||
| `policy-cli` / automation | `policy:read`, `policy:author`, `policy:review`, `policy:simulate`, `findings:read` *(optionally add `policy:approve` / `policy:operate` / `policy:activate` for promotion pipelines)* | Keep scopes minimal; reroll CLI/CI tokens issued before 2025‑10‑27 so they drop legacy scope names and adopt the new set. |
|
||||
| UI/editor sessions | `policy:read`, `policy:author`, `policy:simulate` (+ reviewer/approver/operator scopes as appropriate) | Issue tenant-specific clients so audit and rate limits remain scoped. |
|
||||
|
||||
Sample YAML entry:
|
||||
|
||||
```yaml
|
||||
- clientId: "policy-engine"
|
||||
displayName: "Policy Engine Service"
|
||||
grantTypes: [ "client_credentials" ]
|
||||
audiences: [ "api://policy-engine" ]
|
||||
scopes: [ "policy:run", "findings:read", "effective:write" ]
|
||||
tenant: "tenant-default"
|
||||
properties:
|
||||
serviceIdentity: "policy-engine"
|
||||
senderConstraint: "dpop"
|
||||
auth:
|
||||
type: "client_secret"
|
||||
secretFile: "../secrets/policy-engine.secret"
|
||||
```
|
||||
|
||||
Compliance checklist:
|
||||
|
||||
- [ ] `policy-engine` client includes `properties.serviceIdentity: policy-engine` and a tenant hint; logins missing either are rejected.
|
||||
- [ ] Non-service clients omit `effective:write` and receive only the scopes required for their role (`policy:read`, `policy:author`, `policy:review`, `policy:approve`, `policy:operate`, `policy:simulate`, etc.).
|
||||
- [ ] Legacy tokens using `policy:write`/`policy:submit`/`policy:edit` are rotated to the new scope set before Production change freeze (see release migration note below).
|
||||
- [ ] Approval/activation workflows use identities distinct from authoring identities; tenants are provisioned per client to keep telemetry segregated.
|
||||
- [ ] Operators document reviewer assignments and incident procedures alongside `/docs/security/policy-governance.md` and archive policy evidence bundles (`stella policy bundle export`) with each release.
|
||||
|
||||
### 7.3 Orchestrator roles & scopes
|
||||
|
||||
| Role / Client | Scopes | Notes |
|
||||
| --- | --- | --- |
|
||||
| `Orch.Viewer` role | `orch:read` | Read-only access to Orchestrator dashboards, queues, and telemetry. |
|
||||
| `Orch.Operator` role | `orch:read`, `orch:operate` | Issue short-lived tokens for control actions (pause/resume, retry, sync). Token requests **must** include `operator_reason` (≤256 chars) and `operator_ticket` (≤128 chars); Authority rejects requests missing either value and records both in audit events. |
|
||||
|
||||
Token request example via client credentials:
|
||||
|
||||
```bash
|
||||
curl -u orch-operator:s3cr3t! \
|
||||
-d 'grant_type=client_credentials' \
|
||||
-d 'scope=orch:operate' \
|
||||
-d 'operator_reason=resume source after maintenance' \
|
||||
-d 'operator_ticket=INC-2045' \
|
||||
https://authority.example.com/token
|
||||
```
|
||||
|
||||
Tokens lacking `operator_reason` or `operator_ticket` receive `invalid_request`; audit events (`authority.client_credentials.grant`) surface the supplied values under `request.reason` and `request.ticket` for downstream review.
|
||||
CLI clients set these parameters via `Authority.OperatorReason` / `Authority.OperatorTicket` (environment variables `STELLAOPS_ORCH_REASON` and `STELLAOPS_ORCH_TICKET`).
|
||||
|
||||
## 8. Offline & Sovereign Operation
|
||||
- **No outbound dependencies:** Authority only contacts MongoDB and local plugins. Discovery and JWKS are cached by clients with offline tolerances (`AllowOfflineCacheFallback`, `OfflineCacheTolerance`). Operators should mirror these responses for air-gapped use.
|
||||
- **Structured logging:** Every revocation export, signing rotation, bootstrap action, and token issuance emits structured logs with `traceId`, `client_id`, `subjectId`, and `network.remoteIp` where applicable. Mirror logs to your SIEM to retain audit trails without central connectivity.
|
||||
- **Determinism:** Sorting rules in token and revocation exports guarantee byte-for-byte identical artefacts given the same datastore state. Hashes and signatures remain stable across machines.
|
||||
|
||||
## 9. Operational Checklist
|
||||
- [ ] Protect the bootstrap API key and disable bootstrap endpoints (`bootstrap.enabled: false`) once initial setup is complete.
|
||||
- [ ] Schedule `stella auth revoke export` (or `/internal/revocations/export`) at the same cadence as Concelier exports so bundles remain in lockstep.
|
||||
- [ ] Rotate signing keys before expiration; keep at least one retired key until all cached bundles/tokens signed with it have expired.
|
||||
- [ ] Monitor `/health` and `/ready` plus rate-limiter metrics to detect plugin outages early.
|
||||
- [ ] Ensure downstream services cache JWKS and revocation bundles within tolerances; stale caches risk accepting revoked tokens.
|
||||
|
||||
For plug-in specific requirements, refer to **[Authority Plug-in Developer Guide](dev/31_AUTHORITY_PLUGIN_DEVELOPER_GUIDE.md)**. For revocation bundle validation workflow, see **[Authority Revocation Bundle](security/revocation-bundle.md)**.
|
||||
# StellaOps Authority Service
|
||||
|
||||
> **Status:** Drafted 2025-10-12 (CORE5B.DOC / DOC1.AUTH) – aligns with Authority revocation store, JWKS rotation, and bootstrap endpoints delivered in Sprint 1.
|
||||
|
||||
## 1. Purpose
|
||||
The **StellaOps Authority** service issues OAuth2/OIDC tokens for every StellaOps module (Concelier, Backend, Agent, Zastava) and exposes the policy controls required in sovereign/offline environments. Authority is built as a minimal ASP.NET host that:
|
||||
|
||||
- brokers password, client-credentials, and device-code flows through pluggable identity providers;
|
||||
- persists access/refresh/device tokens in MongoDB with deterministic schemas for replay analysis and air-gapped audit copies;
|
||||
- distributes revocation bundles and JWKS material so downstream services can enforce lockouts without direct database access;
|
||||
- offers bootstrap APIs for first-run provisioning and key rotation without redeploying binaries.
|
||||
|
||||
Authority is deployed alongside Concelier in air-gapped environments and never requires outbound internet access. All trusted metadata (OpenIddict discovery, JWKS, revocation bundles) is cacheable, signed, and reproducible.
|
||||
|
||||
## 2. Component Architecture
|
||||
Authority is composed of five cooperating subsystems:
|
||||
|
||||
1. **Minimal API host** – configures OpenIddict endpoints (`/token`, `/authorize`, `/revoke`, `/jwks`), publishes the OpenAPI contract at `/.well-known/openapi`, and enables structured logging/telemetry. Rate limiting hooks (`AuthorityRateLimiter`) wrap every request.
|
||||
2. **Plugin host** – loads `StellaOps.Authority.Plugin.*.dll` assemblies, applies capability metadata, and exposes password/client provisioning surfaces through dependency injection.
|
||||
3. **Mongo storage** – persists tokens, revocations, bootstrap invites, and plugin state in deterministic collections indexed for offline sync (`authority_tokens`, `authority_revocations`, etc.).
|
||||
4. **Cryptography layer** – `StellaOps.Cryptography` abstractions manage password hashing, signing keys, JWKS export, and detached JWS generation.
|
||||
5. **Offline ops APIs** – internal endpoints under `/internal/*` provide administrative flows (bootstrap users/clients, revocation export) guarded by API keys and deterministic audit events.
|
||||
|
||||
A high-level sequence for password logins:
|
||||
|
||||
```
|
||||
Client -> /token (password grant)
|
||||
-> Rate limiter & audit hooks
|
||||
-> Plugin credential store (Argon2id verification)
|
||||
-> Token persistence (Mongo authority_tokens)
|
||||
-> Response (access/refresh tokens + deterministic claims)
|
||||
```
|
||||
|
||||
## 3. Token Lifecycle & Persistence
|
||||
Authority persists every issued token in MongoDB so operators can audit or revoke without scanning distributed caches.
|
||||
|
||||
- **Collection:** `authority_tokens`
|
||||
- **Key fields:**
|
||||
- `tokenId`, `type` (`access_token`, `refresh_token`, `device_code`, `authorization_code`)
|
||||
- `subjectId`, `clientId`, ordered `scope` array
|
||||
- `tenant` (lower-cased tenant hint from the issuing client, omitted for global clients)
|
||||
|
||||
### Console OIDC client
|
||||
|
||||
- **Client ID**: `console-web`
|
||||
- **Grants**: `authorization_code` (PKCE required), `refresh_token`
|
||||
- **Audience**: `console`
|
||||
- **Scopes**: `openid`, `profile`, `email`, `advisory:read`, `vex:read`, `aoc:verify`, `findings:read`, `orch:read`, `vuln:read`
|
||||
- **Redirect URIs** (defaults): `https://console.stella-ops.local/oidc/callback`
|
||||
- **Post-logout redirect**: `https://console.stella-ops.local/`
|
||||
- **Tokens**: Access tokens inherit the global 2 minute lifetime; refresh tokens remain short-lived (30 days) and can be exchanged silently via `/token`.
|
||||
- **Roles**: Assign Authority role `Orch.Viewer` (exposed to tenants as `role/orch-viewer`) when operators need read-only access to Orchestrator telemetry via Console dashboards. Policy Studio ships dedicated roles (`role/policy-author`, `role/policy-reviewer`, `role/policy-approver`, `role/policy-operator`, `role/policy-auditor`) that align with the new `policy:*` scope family; issue them per tenant so audit trails remain scoped.
|
||||
|
||||
Configuration sample (`etc/authority.yaml.sample`) seeds the client with a confidential secret so Console can negotiate the code exchange on the backend while browsers execute the PKCE dance.
|
||||
|
||||
### Console Authority endpoints
|
||||
|
||||
- `/console/tenants` — Requires `authority:tenants.read`; returns the tenant catalogue for the authenticated principal. Requests lacking the `X-Stella-Tenant` header are rejected (`tenant_header_missing`) and logged.
|
||||
- `/console/profile` — Requires `ui.read`; exposes subject metadata (roles, scopes, audiences) and indicates whether the session is within the five-minute fresh-auth window.
|
||||
- `/console/token/introspect` — Requires `ui.read`; introspects the active access token so the SPA can prompt for re-authentication before privileged actions.
|
||||
|
||||
All endpoints demand DPoP-bound tokens and propagate structured audit events (`authority.console.*`). Gateways must forward the `X-Stella-Tenant` header derived from the access token; downstream services rely on the same value for isolation. Keep Console access tokens short-lived (default 15 minutes) and enforce the fresh-auth window for admin actions (`ui.admin`, `authority:*`, `policy:activate`, `exceptions:approve`).
|
||||
- `status` (`valid`, `revoked`, `expired`), `createdAt`, optional `expiresAt`
|
||||
- `revokedAt`, machine-readable `revokedReason`, optional `revokedReasonDescription`
|
||||
- `revokedMetadata` (string dictionary for plugin-specific context)
|
||||
- **Persistence flow:** `PersistTokensHandler` stamps missing JWT IDs, normalises scopes, and stores every principal emitted by OpenIddict.
|
||||
- **Revocation flow:** `AuthorityTokenStore.UpdateStatusAsync` flips status, records the reason metadata, and is invoked by token revocation handlers and plugin provisioning events (e.g., disabling a user).
|
||||
- **Expiry maintenance:** `AuthorityTokenStore.DeleteExpiredAsync` prunes non-revoked tokens past their `expiresAt` timestamp. Operators should schedule this in maintenance windows if large volumes of tokens are issued.
|
||||
|
||||
### Expectations for resource servers
|
||||
Resource servers (Concelier WebService, Backend, Agent) **must not** assume in-memory caches are authoritative. They should:
|
||||
|
||||
- cache `/jwks` and `/revocations/export` responses within configured lifetimes;
|
||||
- honour `revokedReason` metadata when shaping audit trails;
|
||||
- treat `status != "valid"` or missing tokens as immediate denial conditions.
|
||||
- propagate the `tenant` claim (`X-Stella-Tenant` header in REST calls) and reject requests when the tenant supplied by Authority does not match the resource server's scope; Concelier and Excititor guard endpoints refuse cross-tenant tokens.
|
||||
|
||||
### Tenant propagation
|
||||
|
||||
- Client provisioning (bootstrap or plug-in) accepts a `tenant` hint. Authority normalises the value (`trim().ToLowerInvariant()`) and persists it alongside the registration. Clients without an explicit tenant remain global.
|
||||
- Issued principals include the `stellaops:tenant` claim. `PersistTokensHandler` mirrors this claim into `authority_tokens.tenant`, enabling per-tenant revocation and reporting.
|
||||
- Rate limiter metadata now tags requests with `authority.tenant`, unlocking per-tenant throughput metrics and diagnostic filters. Audit events (`authority.client_credentials.grant`, `authority.password.grant`, bootstrap flows) surface the tenant and login attempt documents index on `{tenant, occurredAt}` for quick queries.
|
||||
- Client credentials that request `advisory:ingest`, `advisory:read`, `vex:ingest`, `vex:read`, `signals:read`, `signals:write`, `signals:admin`, or `aoc:verify` now fail fast when the client registration lacks a tenant hint. Issued tokens are re-validated against persisted tenant metadata, and Authority rejects any cross-tenant replay (`invalid_client`/`invalid_token`), ensuring aggregation-only workloads remain tenant-scoped.
|
||||
- Client credentials that request `export.viewer`, `export.operator`, or `export.admin` must provide a tenant hint. Requests for `export.admin` also need accompanying `export_reason` and `export_ticket` parameters; Authority returns `invalid_request` when either value is missing and records the denial in token audit events.
|
||||
- Policy Studio scopes (`policy:author`, `policy:review`, `policy:approve`, `policy:operate`, `policy:audit`, `policy:simulate`, `policy:run`, `policy:activate`) require a tenant assignment; Authority rejects tokens missing the hint with `invalid_client` and records `scope.invalid` metadata for auditing.
|
||||
- **AOC pairing guardrails** – Tokens that request `advisory:read`, `vex:read`, or any `signals:*` scope must also request `aoc:verify`. Authority rejects mismatches with `invalid_scope` (`Scope 'aoc:verify' is required when requesting advisory/vex read scopes.` or `Scope 'aoc:verify' is required when requesting signals scopes.`) so automation surfaces deterministic errors.
|
||||
- **Signals ingestion guardrails** – Sensors and services requesting `signals:write`/`signals:admin` must also request `aoc:verify`; Authority records the `authority.aoc_scope_violation` tag when the pairing is missing so operators can trace failing sensors immediately.
|
||||
- Password grant flows reuse the client registration's tenant and enforce the configured scope allow-list. Requested scopes outside that list (or mismatched tenants) trigger `invalid_scope`/`invalid_client` failures, ensuring cross-tenant access is denied before token issuance.
|
||||
|
||||
### Default service scopes
|
||||
|
||||
| Client ID | Purpose | Scopes granted | Sender constraint | Tenant |
|
||||
|----------------------|---------------------------------------|--------------------------------------|-------------------|-----------------|
|
||||
| `concelier-ingest` | Concelier raw advisory ingestion | `advisory:ingest`, `advisory:read` | `dpop` | `tenant-default` |
|
||||
| `excitor-ingest` | Excititor raw VEX ingestion | `vex:ingest`, `vex:read` | `dpop` | `tenant-default` |
|
||||
| `aoc-verifier` | Aggregation-only contract verification | `aoc:verify`, `advisory:read`, `vex:read` | `dpop` | `tenant-default` |
|
||||
| `cartographer-service` | Graph snapshot construction | `graph:write`, `graph:read` | `dpop` | `tenant-default` |
|
||||
| `graph-api` | Graph Explorer gateway/API | `graph:read`, `graph:export`, `graph:simulate` | `dpop` | `tenant-default` |
|
||||
| `export-center-operator` | Export Center operator automation | `export.viewer`, `export.operator` | `dpop` | `tenant-default` |
|
||||
| `export-center-admin` | Export Center administrative automation | `export.viewer`, `export.operator`, `export.admin` | `dpop` | `tenant-default` |
|
||||
| `vuln-explorer-ui` | Vuln Explorer UI/API | `vuln:read` | `dpop` | `tenant-default` |
|
||||
| `signals-uploader` | Reachability sensor ingestion | `signals:write`, `signals:read`, `aoc:verify` | `dpop` | `tenant-default` |
|
||||
|
||||
> **Secret hygiene (2025‑10‑27):** The repository includes a convenience `etc/authority.yaml` for compose/helm smoke tests. Every entry’s `secretFile` points to `etc/secrets/*.secret`, which ship with `*-change-me` placeholders—replace them with strong values (and wire them through your vault/secret manager) before issuing tokens in CI, staging, or production.
|
||||
|
||||
For factory provisioning, issue sensors the **SignalsUploader** role template (`signals:write`, `signals:read`, `aoc:verify`). Authority rejects ingestion tokens that omit `aoc:verify`, preserving aggregation-only contract guarantees for reachability signals.
|
||||
|
||||
These registrations are provided as examples in `etc/authority.yaml.sample`. Clone them per tenant (for example `concelier-tenant-a`, `concelier-tenant-b`) so tokens remain tenant-scoped by construction.
|
||||
|
||||
Graph Explorer introduces dedicated scopes: `graph:write` for Cartographer build jobs, `graph:read` for query/read operations, `graph:export` for long-running export downloads, and `graph:simulate` for what-if overlays. Assign only the scopes a client actually needs to preserve least privilege—UI-facing clients should typically request read/export access, while background services (Cartographer, Scheduler) require write privileges.
|
||||
|
||||
#### Least-privilege guidance for graph clients
|
||||
|
||||
- **Service identities** – The Cartographer worker should request `graph:write` and `graph:read` only; grant `graph:simulate` exclusively to pipeline automation that invokes Policy Engine overlays on demand. Keep `graph:export` scoped to API gateway components responsible for streaming GraphML/JSONL artifacts. Authority enforces this by rejecting `graph:write` tokens that lack `properties.serviceIdentity: cartographer`.
|
||||
- **Tenant propagation** – Every client registration must pin a `tenant` hint. Authority normalises the value and stamps it into issued tokens (`stellaops:tenant`) so downstream services (Scheduler, Graph API, Console) can enforce tenant isolation without custom headers. Graph scopes (`graph:read`, `graph:write`, `graph:export`, `graph:simulate`) are denied if the tenant hint is missing.
|
||||
- **SDK alignment** – Use the generated `StellaOpsScopes` constants in service code to request graph scopes. Hard-coded strings risk falling out of sync as additional graph capabilities are added.
|
||||
- **DPOP for automation** – Maintain sender-constrained (`dpop`) flows for Cartographer and Scheduler to limit reuse of access tokens if a build host is compromised. For UI-facing tokens, pair `graph:read`/`graph:export` with short lifetimes and enforce refresh-token rotation at the gateway.
|
||||
|
||||
#### Export Center scope guardrails
|
||||
|
||||
- **Viewer vs operator** – `export.viewer` grants read-only access to export profiles, manifests, and bundles. Automation that schedules or reruns exports should request `export.operator` (and typically `export.viewer`). Tenant hints remain mandatory; Authority refuses tokens without them.
|
||||
- **Administrative mutations** – Changes to retention policies, encryption key references, or schedule defaults require `export.admin`. When requesting tokens with this scope, clients must supply `export_reason` and `export_ticket` parameters; Authority persists the values for audit records and rejects missing metadata with `invalid_request`.
|
||||
- **Operational hygiene** – Rotate `export.admin` credentials infrequently and run them through fresh-auth workflows where possible. Prefer distributing verification tooling with `export.viewer` tokens for day-to-day bundle validation.
|
||||
|
||||
#### Vuln Explorer permalinks
|
||||
|
||||
- **Scope** – `vuln:read` authorises Vuln Explorer to fetch advisory/linkset evidence and issue shareable links. Assign it only to front-end/API clients that must render vulnerability details.
|
||||
- **Signed links** – `POST /permalinks/vuln` (requires `vuln:read`) accepts `{ "tenant": "tenant-a", "resourceKind": "vulnerability", "state": { ... }, "expiresInSeconds": 86400 }` and returns a JWT (`token`) plus `issuedAt`/`expiresAt`. The token embeds the tenant, requested state, and `vuln:read` scope and is signed with the same Authority signing keys published via `/jwks`.
|
||||
- **Validation** – Resource servers verify the permalink using cached JWKS: check signature, ensure the tenant matches the current request context, honour the expiry, and enforce the contained `vuln:read` scope. The payload’s `resource.state` block is opaque JSON so UIs can round-trip filters/search terms without new schema changes.
|
||||
|
||||
## 4. Revocation Pipeline
|
||||
Authority centralises revocation in `authority_revocations` with deterministic categories:
|
||||
|
||||
| Category | Meaning | Required fields |
|
||||
| --- | --- | --- |
|
||||
| `token` | Specific OAuth token revoked early. | `revocationId` (token id), `tokenType`, optional `clientId`, `subjectId` |
|
||||
| `subject` | All tokens for a subject disabled. | `revocationId` (= subject id) |
|
||||
| `client` | OAuth client registration revoked. | `revocationId` (= client id) |
|
||||
| `key` | Signing/JWE key withdrawn. | `revocationId` (= key id) |
|
||||
|
||||
`RevocationBundleBuilder` flattens Mongo documents into canonical JSON, sorts entries by (`category`, `revocationId`, `revokedAt`), and signs exports using detached JWS (RFC 7797) with cosign-compatible headers.
|
||||
|
||||
**Export surfaces** (deterministic output, suitable for Offline Kit):
|
||||
|
||||
- CLI: `stella auth revoke export --output ./out` writes `revocation-bundle.json`, `.jws`, `.sha256`.
|
||||
- Verification: `stella auth revoke verify --bundle <path> --signature <path> --key <path>` validates detached JWS signatures before distribution, selecting the crypto provider advertised in the detached header (see `docs/security/revocation-bundle.md`).
|
||||
- API: `GET /internal/revocations/export` (requires bootstrap API key) returns the same payload.
|
||||
- Verification: `stella auth revoke verify` validates schema, digest, and detached JWS using cached JWKS or offline keys, automatically preferring the hinted provider (libsodium builds honour `provider=libsodium`; other builds fall back to the managed provider).
|
||||
|
||||
**Consumer guidance:**
|
||||
|
||||
1. Mirror `revocation-bundle.json*` alongside Concelier exports. Offline agents fetch both over the existing update channel.
|
||||
2. Use bundle `sequence` and `bundleId` to detect replay or monotonicity regressions. Ignore bundles with older sequence numbers unless `bundleId` changes and `issuedAt` advances.
|
||||
3. Treat `revokedReason` taxonomy as machine-friendly codes (`compromised`, `rotation`, `policy`, `lifecycle`). Translating to human-readable logs is the consumer’s responsibility.
|
||||
|
||||
## 5. Signing Keys & JWKS Rotation
|
||||
Authority signs revocation bundles and publishes JWKS entries via the new signing manager:
|
||||
|
||||
- **Configuration (`authority.yaml`):**
|
||||
```yaml
|
||||
signing:
|
||||
enabled: true
|
||||
algorithm: ES256 # Defaults to ES256
|
||||
keySource: file # Loader identifier (file, vault, etc.)
|
||||
provider: default # Optional preferred crypto provider
|
||||
activeKeyId: authority-signing-dev
|
||||
keyPath: "../certificates/authority-signing-dev.pem"
|
||||
additionalKeys:
|
||||
- keyId: authority-signing-dev-2024
|
||||
path: "../certificates/authority-signing-dev-2024.pem"
|
||||
source: "file"
|
||||
```
|
||||
- **Sources:** The default loader supports PEM files relative to the content root; additional loaders can be registered via `IAuthoritySigningKeySource`.
|
||||
- **Providers:** Keys are registered against the `ICryptoProviderRegistry`, so alternative implementations (HSM, libsodium) can be plugged in without changing host code.
|
||||
- **OpenAPI discovery:** `GET /.well-known/openapi` returns the published authentication contract (JSON by default, YAML when requested). Responses include `X-StellaOps-Service`, `X-StellaOps-Api-Version`, `X-StellaOps-Build-Version`, plus grant and scope headers, and honour conditional requests via `ETag`/`If-None-Match`.
|
||||
- **JWKS output:** `GET /jwks` lists every signing key with `status` metadata (`active`, `retired`). Old keys remain until operators remove them from configuration, allowing verification of historical bundles/tokens.
|
||||
|
||||
### Rotation SOP (no downtime)
|
||||
1. Generate a new P-256 private key (PEM) on an offline workstation and place it where the Authority host can read it (e.g., `../certificates/authority-signing-2025.pem`).
|
||||
2. Call the authenticated admin API:
|
||||
```bash
|
||||
curl -sS -X POST https://authority.example.com/internal/signing/rotate \
|
||||
-H "x-stellaops-bootstrap-key: ${BOOTSTRAP_KEY}" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"keyId": "authority-signing-2025",
|
||||
"location": "../certificates/authority-signing-2025.pem",
|
||||
"source": "file"
|
||||
}'
|
||||
```
|
||||
3. Verify the response reports the previous key as retired and fetch `/jwks` to confirm the new `kid` appears with `status: "active"`.
|
||||
4. Persist the old key path in `signing.additionalKeys` (the rotation API updates in-memory options; rewrite the YAML to match so restarts remain consistent).
|
||||
5. If you prefer automation, trigger the `.gitea/workflows/authority-key-rotation.yml` workflow with the new `keyId`/`keyPath`; it wraps `ops/authority/key-rotation.sh` and reads environment-specific secrets. The older key will be marked `retired` and appended to `signing.additionalKeys`.
|
||||
6. Re-run `stella auth revoke export` so revocation bundles are signed with the new key. Downstream caches should refresh JWKS within their configured lifetime (`StellaOpsAuthorityOptions.Signing` + client cache tolerance).
|
||||
|
||||
The rotation API leverages the same cryptography abstractions as revocation signing; no restart is required and the previous key is marked `retired` but kept available for verification.
|
||||
|
||||
## 6. Bootstrap & Administrative Endpoints
|
||||
Administrative APIs live under `/internal/*` and require the bootstrap API key plus rate-limiter compliance.
|
||||
|
||||
| Endpoint | Method | Description |
|
||||
| --- | --- | --- |
|
||||
| `/internal/users` | `POST` | Provision initial administrative accounts through the registered password-capable plug-in. Emits structured audit events. |
|
||||
| `/internal/clients` | `POST` | Provision OAuth clients (client credentials / device code). |
|
||||
| `/internal/revocations/export` | `GET` | Export revocation bundle + detached JWS + digest. |
|
||||
| `/internal/signing/rotate` | `POST` | Promote a new signing key (see SOP above). Request body accepts `keyId`, `location`, optional `source`, `algorithm`, `provider`, and metadata. |
|
||||
|
||||
All administrative calls emit `AuthEventRecord` entries enriched with correlation IDs, PII tags, and network metadata for offline SOC ingestion.
|
||||
|
||||
> **Tenant hint:** include a `tenant` entry inside `properties` when bootstrapping clients. Authority normalises the value, stores it on the registration, and stamps future tokens/audit events with the tenant.
|
||||
|
||||
### Bootstrap client example
|
||||
|
||||
```jsonc
|
||||
POST /internal/clients
|
||||
{
|
||||
"clientId": "concelier",
|
||||
"confidential": true,
|
||||
"displayName": "Concelier Backend",
|
||||
"allowedGrantTypes": ["client_credentials"],
|
||||
"allowedScopes": ["concelier.jobs.trigger", "advisory:ingest", "advisory:read"],
|
||||
"properties": {
|
||||
"tenant": "tenant-default"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
For environments with multiple tenants, repeat the call per tenant-specific client (e.g. `concelier-tenant-a`, `concelier-tenant-b`) or append suffixes to the client identifier.
|
||||
|
||||
### Aggregation-only verification tokens
|
||||
|
||||
- Issue a dedicated client (e.g. `aoc-verifier`) with the scopes `aoc:verify`, `advisory:read`, and `vex:read` for each tenant that runs guard checks. Authority refuses to mint tokens for these scopes unless the client registration provides a tenant hint.
|
||||
- The CLI (`stella aoc verify --tenant <tenant>`) and Console verification panel both call `/aoc/verify` on Concelier and Excititor. Tokens that omit the tenant claim or present a tenant that does not match the stored registration are rejected with `invalid_client`/`invalid_token`.
|
||||
- Audit: `authority.client_credentials.grant` entries record `scope.invalid="aoc:verify"` when requests are rejected because the tenant hint is missing or mismatched.
|
||||
|
||||
### Exception approvals & routing
|
||||
|
||||
- New scopes `exceptions:read`, `exceptions:write`, and `exceptions:approve` govern access to the exception lifecycle. Map these via tenant roles (`exceptions-service`, `exceptions-approver`) as described in `/docs/security/authority-scopes.md`.
|
||||
- Configure approval routing in `authority.yaml` with declarative templates. Each template exposes an `authorityRouteId` for downstream services (Policy Engine, Console) and an optional `requireMfa` flag:
|
||||
|
||||
```yaml
|
||||
exceptions:
|
||||
routingTemplates:
|
||||
- id: "secops"
|
||||
authorityRouteId: "approvals/secops"
|
||||
requireMfa: true
|
||||
description: "Security Operations approval chain"
|
||||
- id: "governance"
|
||||
authorityRouteId: "approvals/governance"
|
||||
requireMfa: false
|
||||
description: "Non-production waiver review"
|
||||
```
|
||||
|
||||
- Clients requesting exception scopes must include a tenant assignment. Authority rejects client-credential flows that request `exceptions:*` with `invalid_client` and logs `scope.invalid="exceptions:write"` (or the requested scope) in `authority.client_credentials.grant` audit events when the tenant hint is missing.
|
||||
- When any configured routing template sets `requireMfa: true`, user-facing tokens that contain `exceptions:approve` must be acquired through an MFA-capable identity provider. Password/OIDC flows that lack MFA support are rejected with `authority.password.grant` audit events where `reason="Exception approval scope requires an MFA-capable identity provider."`
|
||||
- Update interactive clients (Console) to request `exceptions:read` by default and elevate to `exceptions:approve` only inside fresh-auth workflows for approvers. Documented examples live in `etc/authority.yaml.sample`.
|
||||
- Verification responses map guard failures to `ERR_AOC_00x` codes and Authority emits `authority.client_credentials.grant` + `authority.token.validate_access` audit records containing the tenant and scopes so operators can trace who executed a run.
|
||||
- For air-gapped or offline replicas, pre-issue verification tokens per tenant and rotate them alongside ingest credentials; the guard endpoints never mutate data and remain safe to expose through the offline kit schedule.
|
||||
|
||||
## 7. Configuration Reference
|
||||
|
||||
| Section | Key | Description | Notes |
|
||||
| --- | --- | --- | --- |
|
||||
| Root | `issuer` | Absolute HTTPS issuer advertised to clients. | Required. Loopback HTTP allowed only for development. |
|
||||
| Tokens | `accessTokenLifetime`, `refreshTokenLifetime`, etc. | Lifetimes for each grant (access, refresh, device, authorization code, identity). | Enforced during issuance; persisted on each token document. |
|
||||
| Storage | `storage.connectionString` | MongoDB connection string. | Required even for tests; offline kits ship snapshots for seeding. |
|
||||
| Signing | `signing.enabled` | Enable JWKS/revocation signing. | Disable only for development. |
|
||||
| Signing | `signing.algorithm` | Signing algorithm identifier. | Currently ES256; additional curves can be wired through crypto providers. |
|
||||
| Signing | `signing.keySource` | Loader identifier (`file`, `vault`, custom). | Determines which `IAuthoritySigningKeySource` resolves keys. |
|
||||
| Signing | `signing.keyPath` | Relative/absolute path understood by the loader. | Stored as-is; rotation request should keep it in sync with filesystem layout. |
|
||||
| Signing | `signing.activeKeyId` | Active JWKS / revocation signing key id. | Exposed as `kid` in JWKS and bundles. |
|
||||
| Signing | `signing.additionalKeys[].keyId` | Retired key identifier retained for verification. | Manager updates this automatically after rotation; keep YAML aligned. |
|
||||
| Signing | `signing.additionalKeys[].source` | Loader identifier per retired key. | Defaults to `signing.keySource` if omitted. |
|
||||
| Security | `security.rateLimiting` | Fixed-window limits for `/token`, `/authorize`, `/internal/*`. | See `docs/security/rate-limits.md` for tuning. |
|
||||
| Bootstrap | `bootstrap.apiKey` | Shared secret required for `/internal/*`. | Only required when `bootstrap.enabled` is true. |
|
||||
|
||||
### 7.1 Sender-constrained clients (DPoP & mTLS)
|
||||
|
||||
Authority now understands two flavours of sender-constrained OAuth clients:
|
||||
|
||||
- **DPoP proof-of-possession** – clients sign a `DPoP` header for `/token` requests. Authority validates the JWK thumbprint, HTTP method/URI, and replay window, then stamps the resulting access token with `cnf.jkt` so downstream services can verify the same key is reused.
|
||||
- Configure under `security.senderConstraints.dpop`. `allowedAlgorithms`, `proofLifetime`, and `replayWindow` are enforced at validation time.
|
||||
- `security.senderConstraints.dpop.nonce.enabled` enables nonce challenges for high-value audiences (`requiredAudiences`, normalised to case-insensitive strings). When a nonce is required but missing or expired, `/token` replies with `WWW-Authenticate: DPoP error="use_dpop_nonce"` (and, when available, a fresh `DPoP-Nonce` header). Clients must retry with the issued nonce embedded in the proof.
|
||||
- `security.senderConstraints.dpop.nonce.store` selects `memory` (default) or `redis`. When `redis` is configured, set `security.senderConstraints.dpop.nonce.redisConnectionString` so replicas share nonce issuance and high-value clients avoid replay gaps during failover.
|
||||
- Example (enabling Redis-backed nonces; adjust audiences per deployment):
|
||||
```yaml
|
||||
security:
|
||||
senderConstraints:
|
||||
dpop:
|
||||
enabled: true
|
||||
proofLifetime: "00:02:00"
|
||||
replayWindow: "00:05:00"
|
||||
allowedAlgorithms: [ "ES256", "ES384" ]
|
||||
nonce:
|
||||
enabled: true
|
||||
ttl: "00:10:00"
|
||||
maxIssuancePerMinute: 120
|
||||
store: "redis"
|
||||
redisConnectionString: "redis://authority-redis:6379?ssl=false"
|
||||
requiredAudiences:
|
||||
- "signer"
|
||||
- "attestor"
|
||||
```
|
||||
Operators can override any field via environment variables (e.g. `STELLAOPS_AUTHORITY__SECURITY__SENDERCONSTRAINTS__DPOP__NONCE__STORE=redis`).
|
||||
- Declare client `audiences` in bootstrap manifests or plug-in provisioning metadata; Authority now defaults the token `aud` claim and `resource` indicator from this list, which is also used to trigger nonce enforcement for audiences such as `signer` and `attestor`.
|
||||
- **Mutual TLS clients** – client registrations may declare an mTLS binding (`senderConstraint: mtls`). When enabled via `security.senderConstraints.mtls`, Authority validates the presented client certificate against stored bindings (`certificateBindings[]`), optional chain verification, and timing windows. Successful requests embed `cnf.x5t#S256` into the access token (and introspection output) so resource servers can enforce the certificate thumbprint.
|
||||
- `security.senderConstraints.mtls.enforceForAudiences` forces mTLS whenever the requested `aud`/`resource` (or the client's configured audiences) intersect the configured allow-list (default includes `signer`). Clients configured for different sender constraints are rejected early so operator policy remains consistent.
|
||||
- Certificate bindings now act as an allow-list: Authority verifies thumbprint, subject, issuer, serial number, and any declared SAN values against the presented certificate, with rotation grace windows applied to `notBefore/notAfter`. Operators can enforce subject regexes, SAN type allow-lists (`dns`, `uri`, `ip`), trusted certificate authorities, and rotation grace via `security.senderConstraints.mtls.*`.
|
||||
|
||||
Both modes persist additional metadata in `authority_tokens`: `senderConstraint` records the enforced policy, while `senderKeyThumbprint` stores the DPoP JWK thumbprint or mTLS certificate hash captured at issuance. Downstream services can rely on these fields (and the corresponding `cnf` claim) when auditing offline copies of the token store.
|
||||
|
||||
### 7.2 Policy Engine clients & scopes
|
||||
|
||||
Policy Engine v2 introduces dedicated scopes and a service identity that materialises effective findings. Configure Authority as follows when provisioning policy clients:
|
||||
|
||||
| Client | Scopes | Notes |
|
||||
| --- | --- | --- |
|
||||
| `policy-engine` (service) | `policy:run`, `findings:read`, `effective:write` | Must include `properties.serviceIdentity: policy-engine` and a tenant. Authority rejects `effective:write` tokens without the marker or tenant. |
|
||||
| `policy-cli` / automation | `policy:read`, `policy:author`, `policy:review`, `policy:simulate`, `findings:read` *(optionally add `policy:approve` / `policy:operate` / `policy:activate` for promotion pipelines)* | Keep scopes minimal; reroll CLI/CI tokens issued before 2025‑10‑27 so they drop legacy scope names and adopt the new set. |
|
||||
| UI/editor sessions | `policy:read`, `policy:author`, `policy:simulate` (+ reviewer/approver/operator scopes as appropriate) | Issue tenant-specific clients so audit and rate limits remain scoped. |
|
||||
|
||||
Sample YAML entry:
|
||||
|
||||
```yaml
|
||||
- clientId: "policy-engine"
|
||||
displayName: "Policy Engine Service"
|
||||
grantTypes: [ "client_credentials" ]
|
||||
audiences: [ "api://policy-engine" ]
|
||||
scopes: [ "policy:run", "findings:read", "effective:write" ]
|
||||
tenant: "tenant-default"
|
||||
properties:
|
||||
serviceIdentity: "policy-engine"
|
||||
senderConstraint: "dpop"
|
||||
auth:
|
||||
type: "client_secret"
|
||||
secretFile: "../secrets/policy-engine.secret"
|
||||
```
|
||||
|
||||
Compliance checklist:
|
||||
|
||||
- [ ] `policy-engine` client includes `properties.serviceIdentity: policy-engine` and a tenant hint; logins missing either are rejected.
|
||||
- [ ] Non-service clients omit `effective:write` and receive only the scopes required for their role (`policy:read`, `policy:author`, `policy:review`, `policy:approve`, `policy:operate`, `policy:simulate`, etc.).
|
||||
- [ ] Legacy tokens using `policy:write`/`policy:submit`/`policy:edit` are rotated to the new scope set before Production change freeze (see release migration note below).
|
||||
- [ ] Approval/activation workflows use identities distinct from authoring identities; tenants are provisioned per client to keep telemetry segregated.
|
||||
- [ ] Operators document reviewer assignments and incident procedures alongside `/docs/security/policy-governance.md` and archive policy evidence bundles (`stella policy bundle export`) with each release.
|
||||
|
||||
### 7.3 Orchestrator roles & scopes
|
||||
|
||||
| Role / Client | Scopes | Notes |
|
||||
| --- | --- | --- |
|
||||
| `Orch.Viewer` role | `orch:read` | Read-only access to Orchestrator dashboards, queues, and telemetry. |
|
||||
| `Orch.Operator` role | `orch:read`, `orch:operate` | Issue short-lived tokens for control actions (pause/resume, retry, sync). Token requests **must** include `operator_reason` (≤256 chars) and `operator_ticket` (≤128 chars); Authority rejects requests missing either value and records both in audit events. |
|
||||
|
||||
Token request example via client credentials:
|
||||
|
||||
```bash
|
||||
curl -u orch-operator:s3cr3t! \
|
||||
-d 'grant_type=client_credentials' \
|
||||
-d 'scope=orch:operate' \
|
||||
-d 'operator_reason=resume source after maintenance' \
|
||||
-d 'operator_ticket=INC-2045' \
|
||||
https://authority.example.com/token
|
||||
```
|
||||
|
||||
Tokens lacking `operator_reason` or `operator_ticket` receive `invalid_request`; audit events (`authority.client_credentials.grant`) surface the supplied values under `request.reason` and `request.ticket` for downstream review.
|
||||
CLI clients set these parameters via `Authority.OperatorReason` / `Authority.OperatorTicket` (environment variables `STELLAOPS_ORCH_REASON` and `STELLAOPS_ORCH_TICKET`).
|
||||
|
||||
## 8. Offline & Sovereign Operation
|
||||
- **No outbound dependencies:** Authority only contacts MongoDB and local plugins. Discovery and JWKS are cached by clients with offline tolerances (`AllowOfflineCacheFallback`, `OfflineCacheTolerance`). Operators should mirror these responses for air-gapped use.
|
||||
- **Structured logging:** Every revocation export, signing rotation, bootstrap action, and token issuance emits structured logs with `traceId`, `client_id`, `subjectId`, and `network.remoteIp` where applicable. Mirror logs to your SIEM to retain audit trails without central connectivity.
|
||||
- **Determinism:** Sorting rules in token and revocation exports guarantee byte-for-byte identical artefacts given the same datastore state. Hashes and signatures remain stable across machines.
|
||||
|
||||
## 9. Operational Checklist
|
||||
- [ ] Protect the bootstrap API key and disable bootstrap endpoints (`bootstrap.enabled: false`) once initial setup is complete.
|
||||
- [ ] Schedule `stella auth revoke export` (or `/internal/revocations/export`) at the same cadence as Concelier exports so bundles remain in lockstep.
|
||||
- [ ] Rotate signing keys before expiration; keep at least one retired key until all cached bundles/tokens signed with it have expired.
|
||||
- [ ] Monitor `/health` and `/ready` plus rate-limiter metrics to detect plugin outages early.
|
||||
- [ ] Ensure downstream services cache JWKS and revocation bundles within tolerances; stale caches risk accepting revoked tokens.
|
||||
|
||||
For plug-in specific requirements, refer to **[Authority Plug-in Developer Guide](dev/31_AUTHORITY_PLUGIN_DEVELOPER_GUIDE.md)**. For revocation bundle validation workflow, see **[Authority Revocation Bundle](security/revocation-bundle.md)**.
|
||||
|
||||
@@ -1,91 +1,91 @@
|
||||
# Data Schemas & Persistence Contracts
|
||||
|
||||
*Audience* – backend developers, plug‑in authors, DB admins.
|
||||
*Scope* – describes **Redis**, **MongoDB** (optional), and on‑disk blob shapes that power Stella Ops.
|
||||
|
||||
---
|
||||
|
||||
## 0 Document Conventions
|
||||
|
||||
* **CamelCase** for JSON.
|
||||
* All timestamps are **RFC 3339 / ISO 8601** with `Z` (UTC).
|
||||
* `⭑` = planned but *not* shipped yet (kept on Feature Matrix “To Do”).
|
||||
|
||||
---
|
||||
|
||||
## 1 SBOM Wrapper Envelope
|
||||
|
||||
Every SBOM blob (regardless of format) is stored on disk or in object storage with a *sidecar* JSON file that indexes it for the scanners.
|
||||
|
||||
#### 1.1 JSON Shape
|
||||
|
||||
```jsonc
|
||||
{
|
||||
"id": "sha256:417f…", // digest of the SBOM *file* itself
|
||||
"imageDigest": "sha256:e2b9…", // digest of the original container image
|
||||
"created": "2025-07-14T07:02:13Z",
|
||||
"format": "trivy-json-v2", // NEW enum: trivy-json-v2 | spdx-json | cyclonedx-json
|
||||
"layers": [
|
||||
"sha256:d38b…", // layer digests (ordered)
|
||||
"sha256:af45…"
|
||||
],
|
||||
"partial": false, // true => delta SBOM (only some layers)
|
||||
"provenanceId": "prov_0291" // ⭑ link to SLSA attestation (Q1‑2026)
|
||||
}
|
||||
```
|
||||
|
||||
*`format`* **NEW** – added to support **multiple SBOM formats**.
|
||||
*`partial`* **NEW** – true when generated via the **delta SBOM** flow (§1.3).
|
||||
|
||||
#### 1.2 File‑system Layout
|
||||
|
||||
```
|
||||
blobs/
|
||||
├─ 417f… # digest prefix
|
||||
│ ├─ sbom.json # payload (any format)
|
||||
│ └─ sbom.meta.json # wrapper (shape above)
|
||||
```
|
||||
|
||||
> **Note** – blob storage can point at S3, MinIO, or plain disk; driver plug‑ins adapt.
|
||||
|
||||
#### 1.3 Delta SBOM Extension
|
||||
|
||||
When `partial: true`, *only* the missing layers have been scanned.
|
||||
Merging logic inside `scanning` module stitches new data onto the cached full SBOM in Redis.
|
||||
|
||||
---
|
||||
|
||||
## 2 Redis Keyspace
|
||||
|
||||
| Key pattern | Type | TTL | Purpose |
|
||||
|-------------------------------------|---------|------|--------------------------------------------------|
|
||||
| `scan:<digest>` | string | ∞ | Last scan JSON result (as returned by `/scan`) |
|
||||
| `layers:<digest>` | set | 90d | Layers already possessing SBOMs (delta cache) |
|
||||
| `policy:active` | string | ∞ | YAML **or** Rego ruleset |
|
||||
| `quota:<token>` | string | *until next UTC midnight* | Per‑token scan counter for Free tier ({{ quota_token }} scans). |
|
||||
| `policy:history` | list | ∞ | Change audit IDs (see Mongo) |
|
||||
| `feed:nvd:json` | string | 24h | Normalised feed snapshot |
|
||||
| `locator:<imageDigest>` | string | 30d | Maps image digest → sbomBlobId |
|
||||
| `metrics:…` | various | — | Prom / OTLP runtime metrics |
|
||||
|
||||
> **Delta SBOM** uses `layers:*` to skip work in <20 ms.
|
||||
> **Quota enforcement** increments `quota:<token>` atomically; when {{ quota_token }} the API returns **429**.
|
||||
|
||||
---
|
||||
|
||||
## 3 MongoDB Collections (Optional)
|
||||
|
||||
Only enabled when `MONGO_URI` is supplied (for long‑term audit).
|
||||
|
||||
| Collection | Shape (summary) | Indexes |
|
||||
|--------------------|------------------------------------------------------------|-------------------------------------|
|
||||
| `sbom_history` | Wrapper JSON + `replaceTs` on overwrite | `{imageDigest}` `{created}` |
|
||||
| `policy_versions` | `{_id, yaml, rego, authorId, created}` | `{created}` |
|
||||
| `attestations` ⭑ | SLSA provenance doc + Rekor log pointer | `{imageDigest}` |
|
||||
| `audit_log` | Fully rendered RFC 5424 entries (UI & CLI actions) | `{userId}` `{ts}` |
|
||||
|
||||
Schema detail for **policy_versions**:
|
||||
|
||||
# Data Schemas & Persistence Contracts
|
||||
|
||||
*Audience* – backend developers, plug‑in authors, DB admins.
|
||||
*Scope* – describes **Redis**, **MongoDB** (optional), and on‑disk blob shapes that power Stella Ops.
|
||||
|
||||
---
|
||||
|
||||
## 0 Document Conventions
|
||||
|
||||
* **CamelCase** for JSON.
|
||||
* All timestamps are **RFC 3339 / ISO 8601** with `Z` (UTC).
|
||||
* `⭑` = planned but *not* shipped yet (kept on Feature Matrix “To Do”).
|
||||
|
||||
---
|
||||
|
||||
## 1 SBOM Wrapper Envelope
|
||||
|
||||
Every SBOM blob (regardless of format) is stored on disk or in object storage with a *sidecar* JSON file that indexes it for the scanners.
|
||||
|
||||
#### 1.1 JSON Shape
|
||||
|
||||
```jsonc
|
||||
{
|
||||
"id": "sha256:417f…", // digest of the SBOM *file* itself
|
||||
"imageDigest": "sha256:e2b9…", // digest of the original container image
|
||||
"created": "2025-07-14T07:02:13Z",
|
||||
"format": "trivy-json-v2", // NEW enum: trivy-json-v2 | spdx-json | cyclonedx-json
|
||||
"layers": [
|
||||
"sha256:d38b…", // layer digests (ordered)
|
||||
"sha256:af45…"
|
||||
],
|
||||
"partial": false, // true => delta SBOM (only some layers)
|
||||
"provenanceId": "prov_0291" // ⭑ link to SLSA attestation (Q1‑2026)
|
||||
}
|
||||
```
|
||||
|
||||
*`format`* **NEW** – added to support **multiple SBOM formats**.
|
||||
*`partial`* **NEW** – true when generated via the **delta SBOM** flow (§1.3).
|
||||
|
||||
#### 1.2 File‑system Layout
|
||||
|
||||
```
|
||||
blobs/
|
||||
├─ 417f… # digest prefix
|
||||
│ ├─ sbom.json # payload (any format)
|
||||
│ └─ sbom.meta.json # wrapper (shape above)
|
||||
```
|
||||
|
||||
> **Note** – blob storage can point at S3, MinIO, or plain disk; driver plug‑ins adapt.
|
||||
|
||||
#### 1.3 Delta SBOM Extension
|
||||
|
||||
When `partial: true`, *only* the missing layers have been scanned.
|
||||
Merging logic inside `scanning` module stitches new data onto the cached full SBOM in Redis.
|
||||
|
||||
---
|
||||
|
||||
## 2 Redis Keyspace
|
||||
|
||||
| Key pattern | Type | TTL | Purpose |
|
||||
|-------------------------------------|---------|------|--------------------------------------------------|
|
||||
| `scan:<digest>` | string | ∞ | Last scan JSON result (as returned by `/scan`) |
|
||||
| `layers:<digest>` | set | 90d | Layers already possessing SBOMs (delta cache) |
|
||||
| `policy:active` | string | ∞ | YAML **or** Rego ruleset |
|
||||
| `quota:<token>` | string | *until next UTC midnight* | Per‑token scan counter for Free tier ({{ quota_token }} scans). |
|
||||
| `policy:history` | list | ∞ | Change audit IDs (see Mongo) |
|
||||
| `feed:nvd:json` | string | 24h | Normalised feed snapshot |
|
||||
| `locator:<imageDigest>` | string | 30d | Maps image digest → sbomBlobId |
|
||||
| `metrics:…` | various | — | Prom / OTLP runtime metrics |
|
||||
|
||||
> **Delta SBOM** uses `layers:*` to skip work in <20 ms.
|
||||
> **Quota enforcement** increments `quota:<token>` atomically; when {{ quota_token }} the API returns **429**.
|
||||
|
||||
---
|
||||
|
||||
## 3 MongoDB Collections (Optional)
|
||||
|
||||
Only enabled when `MONGO_URI` is supplied (for long‑term audit).
|
||||
|
||||
| Collection | Shape (summary) | Indexes |
|
||||
|--------------------|------------------------------------------------------------|-------------------------------------|
|
||||
| `sbom_history` | Wrapper JSON + `replaceTs` on overwrite | `{imageDigest}` `{created}` |
|
||||
| `policy_versions` | `{_id, yaml, rego, authorId, created}` | `{created}` |
|
||||
| `attestations` ⭑ | SLSA provenance doc + Rekor log pointer | `{imageDigest}` |
|
||||
| `audit_log` | Fully rendered RFC 5424 entries (UI & CLI actions) | `{userId}` `{ts}` |
|
||||
|
||||
Schema detail for **policy_versions**:
|
||||
|
||||
Samples live under `samples/api/scheduler/` (e.g., `schedule.json`, `run.json`, `impact-set.json`, `audit.json`) and mirror the canonical serializer output shown below.
|
||||
|
||||
```jsonc
|
||||
@@ -327,34 +327,34 @@ Materialized view powering the Scheduler UI dashboards. Stores the latest roll-u
|
||||
- Schedulers should call the projection service after every run state change so the cache mirrors planner/runner progress.
|
||||
|
||||
Sample file: `samples/api/scheduler/run-summary.json`.
|
||||
|
||||
---
|
||||
|
||||
## 4 Policy Schema (YAML v1.0)
|
||||
|
||||
Minimal viable grammar (subset of OSV‑SCHEMA ideas).
|
||||
|
||||
```yaml
|
||||
version: "1.0"
|
||||
rules:
|
||||
- name: Block Critical
|
||||
severity: [Critical]
|
||||
action: block
|
||||
- name: Ignore Low Dev
|
||||
severity: [Low, None]
|
||||
environments: [dev, staging]
|
||||
action: ignore
|
||||
expires: "2026-01-01"
|
||||
- name: Escalate RegionalFeed High
|
||||
sources: [NVD, CNNVD, CNVD, ENISA, JVN, BDU]
|
||||
severity: [High, Critical]
|
||||
action: escalate
|
||||
```
|
||||
|
||||
|
||||
---
|
||||
|
||||
## 4 Policy Schema (YAML v1.0)
|
||||
|
||||
Minimal viable grammar (subset of OSV‑SCHEMA ideas).
|
||||
|
||||
```yaml
|
||||
version: "1.0"
|
||||
rules:
|
||||
- name: Block Critical
|
||||
severity: [Critical]
|
||||
action: block
|
||||
- name: Ignore Low Dev
|
||||
severity: [Low, None]
|
||||
environments: [dev, staging]
|
||||
action: ignore
|
||||
expires: "2026-01-01"
|
||||
- name: Escalate RegionalFeed High
|
||||
sources: [NVD, CNNVD, CNVD, ENISA, JVN, BDU]
|
||||
severity: [High, Critical]
|
||||
action: escalate
|
||||
```
|
||||
|
||||
Validation is performed by `policy:mapping.yaml` JSON‑Schema embedded in backend.
|
||||
|
||||
Canonical schema source: `src/StellaOps.Policy/Schemas/policy-schema@1.json` (embedded into `StellaOps.Policy`).
|
||||
`PolicyValidationCli` (see `src/StellaOps.Policy/PolicyValidationCli.cs`) provides the reusable command handler that the main CLI wires up; in the interim it can be invoked from a short host like:
|
||||
Canonical schema source: `src/Policy/__Libraries/StellaOps.Policy/Schemas/policy-schema@1.json` (embedded into `StellaOps.Policy`).
|
||||
`PolicyValidationCli` (see `src/Policy/__Libraries/StellaOps.Policy/PolicyValidationCli.cs`) provides the reusable command handler that the main CLI wires up; in the interim it can be invoked from a short host like:
|
||||
|
||||
```csharp
|
||||
await new PolicyValidationCli().RunAsync(new PolicyValidationCliOptions
|
||||
@@ -363,7 +363,7 @@ await new PolicyValidationCli().RunAsync(new PolicyValidationCliOptions
|
||||
Strict = true,
|
||||
});
|
||||
```
|
||||
|
||||
|
||||
### 4.1 Rego Variant (Advanced – TODO)
|
||||
|
||||
*Accepted but stored as‑is in `rego` field.*
|
||||
@@ -372,7 +372,7 @@ Evaluated via internal **OPA** side‑car once feature graduates from TODO list.
|
||||
### 4.2 Policy Scoring Config (JSON)
|
||||
|
||||
*Schema id.* `https://schemas.stella-ops.org/policy/policy-scoring-schema@1.json`
|
||||
*Source.* `src/StellaOps.Policy/Schemas/policy-scoring-schema@1.json` (embedded in `StellaOps.Policy`), default fixture at `src/StellaOps.Policy/Schemas/policy-scoring-default.json`.
|
||||
*Source.* `src/Policy/__Libraries/StellaOps.Policy/Schemas/policy-scoring-schema@1.json` (embedded in `StellaOps.Policy`), default fixture at `src/Policy/__Libraries/StellaOps.Policy/Schemas/policy-scoring-default.json`.
|
||||
|
||||
```jsonc
|
||||
{
|
||||
@@ -426,25 +426,25 @@ npx ajv validate --spec=draft2020 -c ajv-formats \
|
||||
Planned for Q1‑2026 (kept here for early plug‑in authors).
|
||||
|
||||
```jsonc
|
||||
{
|
||||
"id": "prov_0291",
|
||||
"imageDigest": "sha256:e2b9…",
|
||||
"buildType": "https://slsa.dev/container/v1",
|
||||
"builder": {
|
||||
"id": "https://git.stella-ops.ru/ci/stella-runner@sha256:f7b7…"
|
||||
},
|
||||
"metadata": {
|
||||
"invocation": {
|
||||
"parameters": {"GIT_SHA": "f6a1…"},
|
||||
"buildStart": "2025-07-14T06:59:17Z",
|
||||
"buildEnd": "2025-07-14T07:01:22Z"
|
||||
},
|
||||
"completeness": {"parameters": true}
|
||||
},
|
||||
"materials": [
|
||||
{"uri": "git+https://git…", "digest": {"sha1": "f6a1…"}}
|
||||
],
|
||||
"rekorLogIndex": 99817 // entry in local Rekor mirror
|
||||
{
|
||||
"id": "prov_0291",
|
||||
"imageDigest": "sha256:e2b9…",
|
||||
"buildType": "https://slsa.dev/container/v1",
|
||||
"builder": {
|
||||
"id": "https://git.stella-ops.ru/ci/stella-runner@sha256:f7b7…"
|
||||
},
|
||||
"metadata": {
|
||||
"invocation": {
|
||||
"parameters": {"GIT_SHA": "f6a1…"},
|
||||
"buildStart": "2025-07-14T06:59:17Z",
|
||||
"buildEnd": "2025-07-14T07:01:22Z"
|
||||
},
|
||||
"completeness": {"parameters": true}
|
||||
},
|
||||
"materials": [
|
||||
{"uri": "git+https://git…", "digest": {"sha1": "f6a1…"}}
|
||||
],
|
||||
"rekorLogIndex": 99817 // entry in local Rekor mirror
|
||||
}
|
||||
```
|
||||
|
||||
@@ -509,42 +509,42 @@ done
|
||||
```
|
||||
|
||||
Integration tests can embed the sample fixtures to guarantee deterministic serialisation from the `StellaOps.Notify.Models` DTOs introduced in Sprint 15.
|
||||
|
||||
---
|
||||
|
||||
## 6 Validator Contracts
|
||||
|
||||
* For SBOM wrapper – `ISbomValidator` (DLL plug‑in) must return *typed* error list.
|
||||
* For YAML policies – JSON‑Schema at `/schemas/policy‑v1.json`.
|
||||
* For Rego – OPA `opa eval --fail-defined` under the hood.
|
||||
* For **Free‑tier quotas** – `IQuotaService` integration tests ensure `quota:<token>` resets at UTC midnight and produces correct `Retry‑After` headers.
|
||||
|
||||
---
|
||||
|
||||
## 7 Migration Notes
|
||||
|
||||
1. **Add `format` column** to existing SBOM wrappers; default to `trivy-json-v2`.
|
||||
2. **Populate `layers` & `partial`** via backfill script (ship with `stellopsctl migrate` wizard).
|
||||
3. Policy YAML previously stored in Redis → copy to Mongo if persistence enabled.
|
||||
4. Prepare `attestations` collection (empty) – safe to create in advance.
|
||||
|
||||
---
|
||||
|
||||
## 8 Open Questions / Future Work
|
||||
|
||||
* How to de‑duplicate *identical* Rego policies differing only in whitespace?
|
||||
* Embed *GOST 34.11‑2018* digests when users enable Russian crypto suite?
|
||||
* Should enterprise tiers share the same Redis quota keys or switch to JWT claim `tier != Free` bypass?
|
||||
* Evaluate sliding‑window quota instead of strict daily reset.
|
||||
* Consider rate‑limit for `/layers/missing` to avoid brute‑force enumeration.
|
||||
|
||||
---
|
||||
|
||||
## 9 Change Log
|
||||
|
||||
| Date | Note |
|
||||
|------------|--------------------------------------------------------------------------------|
|
||||
| 2025‑07‑14 | **Added:** `format`, `partial`, delta cache keys, YAML policy schema v1.0. |
|
||||
| 2025‑07‑12 | **Initial public draft** – SBOM wrapper, Redis keyspace, audit collections. |
|
||||
|
||||
---
|
||||
|
||||
---
|
||||
|
||||
## 6 Validator Contracts
|
||||
|
||||
* For SBOM wrapper – `ISbomValidator` (DLL plug‑in) must return *typed* error list.
|
||||
* For YAML policies – JSON‑Schema at `/schemas/policy‑v1.json`.
|
||||
* For Rego – OPA `opa eval --fail-defined` under the hood.
|
||||
* For **Free‑tier quotas** – `IQuotaService` integration tests ensure `quota:<token>` resets at UTC midnight and produces correct `Retry‑After` headers.
|
||||
|
||||
---
|
||||
|
||||
## 7 Migration Notes
|
||||
|
||||
1. **Add `format` column** to existing SBOM wrappers; default to `trivy-json-v2`.
|
||||
2. **Populate `layers` & `partial`** via backfill script (ship with `stellopsctl migrate` wizard).
|
||||
3. Policy YAML previously stored in Redis → copy to Mongo if persistence enabled.
|
||||
4. Prepare `attestations` collection (empty) – safe to create in advance.
|
||||
|
||||
---
|
||||
|
||||
## 8 Open Questions / Future Work
|
||||
|
||||
* How to de‑duplicate *identical* Rego policies differing only in whitespace?
|
||||
* Embed *GOST 34.11‑2018* digests when users enable Russian crypto suite?
|
||||
* Should enterprise tiers share the same Redis quota keys or switch to JWT claim `tier != Free` bypass?
|
||||
* Evaluate sliding‑window quota instead of strict daily reset.
|
||||
* Consider rate‑limit for `/layers/missing` to avoid brute‑force enumeration.
|
||||
|
||||
---
|
||||
|
||||
## 9 Change Log
|
||||
|
||||
| Date | Note |
|
||||
|------------|--------------------------------------------------------------------------------|
|
||||
| 2025‑07‑14 | **Added:** `format`, `partial`, delta cache keys, YAML policy schema v1.0. |
|
||||
| 2025‑07‑12 | **Initial public draft** – SBOM wrapper, Redis keyspace, audit collections. |
|
||||
|
||||
---
|
||||
|
||||
@@ -56,7 +56,7 @@
|
||||
## 3 Test Harness
|
||||
|
||||
* **Runner** – `perf/run.sh`, accepts `--phase` and `--samples`.
|
||||
* **Language analyzers microbench** – `dotnet run --project src/StellaOps.Bench/Scanner.Analyzers/StellaOps.Bench.ScannerAnalyzers/StellaOps.Bench.ScannerAnalyzers.csproj -- --repo-root . --out src/StellaOps.Bench/Scanner.Analyzers/baseline.csv --json out/bench/scanner-analyzers/latest.json --prom out/bench/scanner-analyzers/latest.prom --commit $(git rev-parse HEAD)` produces CSV + JSON + Prometheus gauges for analyzer scenarios. Runs fail if `max_ms` regresses ≥ 20 % against `baseline.csv` or if thresholds are exceeded.
|
||||
* **Language analyzers microbench** – `dotnet run --project src/Bench/StellaOps.Bench/Scanner.Analyzers/StellaOps.Bench.ScannerAnalyzers/StellaOps.Bench.ScannerAnalyzers.csproj -- --repo-root . --out src/Bench/StellaOps.Bench/Scanner.Analyzers/baseline.csv --json out/bench/scanner-analyzers/latest.json --prom out/bench/scanner-analyzers/latest.prom --commit $(git rev-parse HEAD)` produces CSV + JSON + Prometheus gauges for analyzer scenarios. Runs fail if `max_ms` regresses ≥ 20 % against `baseline.csv` or if thresholds are exceeded.
|
||||
* **Metrics** – Prometheus + `jq` extracts; aggregated via `scripts/aggregate.ts`.
|
||||
* **CI** – GitLab CI job *benchmark* publishes JSON to `bench‑artifacts/`.
|
||||
* **Visualisation** – Grafana dashboard *Stella‑Perf* (provisioned JSON).
|
||||
|
||||
@@ -1,47 +1,47 @@
|
||||
# Automated Test‑Suite Overview
|
||||
|
||||
This document enumerates **every automated check** executed by the Stella Ops
|
||||
CI pipeline, from unit level to chaos experiments. It is intended for
|
||||
contributors who need to extend coverage or diagnose failures.
|
||||
|
||||
> **Build parameters** – values such as `{{ dotnet }}` (runtime) and
|
||||
> `{{ angular }}` (UI framework) are injected at build time.
|
||||
|
||||
---
|
||||
|
||||
## Layer map
|
||||
|
||||
| Layer | Tooling | Entry‑point | Frequency |
|
||||
|-------|---------|-------------|-----------|
|
||||
| **1. Unit** | `xUnit` (<code>dotnet test</code>) | `*.Tests.csproj` | per PR / push |
|
||||
| **2. Property‑based** | `FsCheck` | `SbomPropertyTests` | per PR |
|
||||
| **3. Integration (API)** | `Testcontainers` suite | `test/Api.Integration` | per PR + nightly |
|
||||
# Automated Test‑Suite Overview
|
||||
|
||||
This document enumerates **every automated check** executed by the Stella Ops
|
||||
CI pipeline, from unit level to chaos experiments. It is intended for
|
||||
contributors who need to extend coverage or diagnose failures.
|
||||
|
||||
> **Build parameters** – values such as `{{ dotnet }}` (runtime) and
|
||||
> `{{ angular }}` (UI framework) are injected at build time.
|
||||
|
||||
---
|
||||
|
||||
## Layer map
|
||||
|
||||
| Layer | Tooling | Entry‑point | Frequency |
|
||||
|-------|---------|-------------|-----------|
|
||||
| **1. Unit** | `xUnit` (<code>dotnet test</code>) | `*.Tests.csproj` | per PR / push |
|
||||
| **2. Property‑based** | `FsCheck` | `SbomPropertyTests` | per PR |
|
||||
| **3. Integration (API)** | `Testcontainers` suite | `test/Api.Integration` | per PR + nightly |
|
||||
| **4. Integration (DB-merge)** | in-memory Mongo + Redis | `Concelier.Integration` (vulnerability ingest/merge/export service) | per PR |
|
||||
| **5. Contract (gRPC)** | `Buf breaking` | `buf.yaml` files | per PR |
|
||||
| **6. Front‑end unit** | `Jest` | `ui/src/**/*.spec.ts` | per PR |
|
||||
| **7. Front‑end E2E** | `Playwright` | `ui/e2e/**` | nightly |
|
||||
| **8. Lighthouse perf / a11y** | `lighthouse-ci` (Chrome headless) | `ui/dist/index.html` | nightly |
|
||||
| **9. Load** | `k6` scripted scenarios | `k6/*.js` | nightly |
|
||||
| **10. Chaos CPU / OOM** | `pumba` | Docker Compose overlay | weekly |
|
||||
| **11. Dependency scanning** | `Trivy fs` + `dotnet list package --vuln` | root | per PR |
|
||||
| **12. License compliance** | `LicenceFinder` | root | per PR |
|
||||
| **13. SBOM reproducibility** | `in‑toto attestation` diff | GitLab job | release tags |
|
||||
|
||||
---
|
||||
|
||||
## Quality gates
|
||||
|
||||
| Metric | Budget | Gate |
|
||||
|--------|--------|------|
|
||||
| API unit coverage | ≥ 85 % lines | PR merge |
|
||||
| API response P95 | ≤ 120 ms | nightly alert |
|
||||
| Δ‑SBOM warm scan P95 (4 vCPU) | ≤ 5 s | nightly alert |
|
||||
| Lighthouse performance score | ≥ 90 | nightly alert |
|
||||
| Lighthouse accessibility score | ≥ 95 | nightly alert |
|
||||
| k6 sustained RPS drop | < 5 % vs baseline | nightly alert |
|
||||
|
||||
---
|
||||
|
||||
| **5. Contract (gRPC)** | `Buf breaking` | `buf.yaml` files | per PR |
|
||||
| **6. Front‑end unit** | `Jest` | `ui/src/**/*.spec.ts` | per PR |
|
||||
| **7. Front‑end E2E** | `Playwright` | `ui/e2e/**` | nightly |
|
||||
| **8. Lighthouse perf / a11y** | `lighthouse-ci` (Chrome headless) | `ui/dist/index.html` | nightly |
|
||||
| **9. Load** | `k6` scripted scenarios | `k6/*.js` | nightly |
|
||||
| **10. Chaos CPU / OOM** | `pumba` | Docker Compose overlay | weekly |
|
||||
| **11. Dependency scanning** | `Trivy fs` + `dotnet list package --vuln` | root | per PR |
|
||||
| **12. License compliance** | `LicenceFinder` | root | per PR |
|
||||
| **13. SBOM reproducibility** | `in‑toto attestation` diff | GitLab job | release tags |
|
||||
|
||||
---
|
||||
|
||||
## Quality gates
|
||||
|
||||
| Metric | Budget | Gate |
|
||||
|--------|--------|------|
|
||||
| API unit coverage | ≥ 85 % lines | PR merge |
|
||||
| API response P95 | ≤ 120 ms | nightly alert |
|
||||
| Δ‑SBOM warm scan P95 (4 vCPU) | ≤ 5 s | nightly alert |
|
||||
| Lighthouse performance score | ≥ 90 | nightly alert |
|
||||
| Lighthouse accessibility score | ≥ 95 | nightly alert |
|
||||
| k6 sustained RPS drop | < 5 % vs baseline | nightly alert |
|
||||
|
||||
---
|
||||
|
||||
## Local runner
|
||||
|
||||
```bash
|
||||
@@ -63,13 +63,13 @@ The script spins up MongoDB/Redis via Testcontainers and requires:
|
||||
|
||||
The Concelier connector suite includes a regression test (`OsvGhsaParityRegressionTests`)
|
||||
that checks a curated set of GHSA identifiers against OSV responses. The fixture
|
||||
snapshots live in `src/StellaOps.Concelier.Connector.Osv.Tests/Fixtures/` and are kept
|
||||
snapshots live in `src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Osv.Tests/Fixtures/` and are kept
|
||||
deterministic so the parity report remains reproducible.
|
||||
|
||||
To refresh the fixtures when GHSA/OSV payloads change:
|
||||
|
||||
1. Ensure outbound HTTPS access to `https://api.osv.dev` and `https://api.github.com`.
|
||||
2. Run `UPDATE_PARITY_FIXTURES=1 dotnet test src/StellaOps.Concelier.Connector.Osv.Tests/StellaOps.Concelier.Connector.Osv.Tests.csproj`.
|
||||
2. Run `UPDATE_PARITY_FIXTURES=1 dotnet test src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Osv.Tests/StellaOps.Concelier.Connector.Osv.Tests.csproj`.
|
||||
3. Commit the regenerated `osv-ghsa.*.json` files that the test emits (raw snapshots and canonical advisories).
|
||||
|
||||
The regen flow logs `[Parity]` messages and normalises `recordedAt` timestamps so the
|
||||
@@ -82,28 +82,28 @@ fixtures stay stable across machines.
|
||||
```mermaid
|
||||
flowchart LR
|
||||
subgraph fast-path
|
||||
U[xUnit] --> P[FsCheck] --> I1[Testcontainer API]
|
||||
end
|
||||
|
||||
I1 --> FE[Jest]
|
||||
FE --> E2E[Playwright]
|
||||
E2E --> Lighthouse
|
||||
U[xUnit] --> P[FsCheck] --> I1[Testcontainer API]
|
||||
end
|
||||
|
||||
I1 --> FE[Jest]
|
||||
FE --> E2E[Playwright]
|
||||
E2E --> Lighthouse
|
||||
Lighthouse --> INTEG2[Concelier]
|
||||
INTEG2 --> LOAD[k6]
|
||||
LOAD --> CHAOS[pumba]
|
||||
CHAOS --> RELEASE[Attestation diff]
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Adding a new test layer
|
||||
|
||||
1. Extend `scripts/dev-test.sh` so local contributors get the layer by default.
|
||||
2. Add a dedicated GitLab job in `.gitlab-ci.yml` (stage `test` or `nightly`).
|
||||
3. Register the job in `docs/19_TEST_SUITE_OVERVIEW.md` *and* list its metric
|
||||
in `docs/metrics/README.md`.
|
||||
|
||||
---
|
||||
|
||||
*Last updated {{ "now" | date: "%Y‑%m‑%d" }}*
|
||||
|
||||
INTEG2 --> LOAD[k6]
|
||||
LOAD --> CHAOS[pumba]
|
||||
CHAOS --> RELEASE[Attestation diff]
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Adding a new test layer
|
||||
|
||||
1. Extend `scripts/dev-test.sh` so local contributors get the layer by default.
|
||||
2. Add a dedicated GitLab job in `.gitlab-ci.yml` (stage `test` or `nightly`).
|
||||
3. Register the job in `docs/19_TEST_SUITE_OVERVIEW.md` *and* list its metric
|
||||
in `docs/metrics/README.md`.
|
||||
|
||||
---
|
||||
|
||||
*Last updated {{ "now" | date: "%Y‑%m‑%d" }}*
|
||||
|
||||
|
||||
@@ -1,190 +1,190 @@
|
||||
# Stella Ops — Installation Guide (Docker & Air‑Gap)
|
||||
|
||||
<!--
|
||||
This file is processed by the Eleventy build.
|
||||
Do **not** hard‑code versions or quota numbers; inherit from
|
||||
docs/_includes/CONSTANTS.md instead.
|
||||
{{ dotnet }} → ".NET 10 LTS"
|
||||
{{ angular }} → "20"
|
||||
-->
|
||||
|
||||
> **Status — public α not yet published.**
|
||||
> The commands below will work as soon as the first image is tagged
|
||||
> `registry.stella-ops.org/stella-ops/stella-ops:0.1.0-alpha`
|
||||
> (target date: **late 2025**). Track progress on the
|
||||
> [road‑map](/roadmap/).
|
||||
|
||||
---
|
||||
|
||||
## 0 · Prerequisites
|
||||
|
||||
| Item | Minimum | Notes |
|
||||
|------|---------|-------|
|
||||
| Linux | Ubuntu 22.04 LTS / Alma 9 | x86‑64 or arm64 |
|
||||
| CPU / RAM | 2 vCPU / 2 GiB | Laptop baseline |
|
||||
| Disk | 10 GiB SSD | SBOM + vuln DB cache |
|
||||
| Docker | **Engine 25 + Compose v2** | `docker -v` |
|
||||
| TLS | OpenSSL 1.1 + | Self‑signed cert generated at first run |
|
||||
|
||||
---
|
||||
|
||||
## 1 · Connected‑host install (Docker Compose)
|
||||
|
||||
```bash
|
||||
# 1. Make a working directory
|
||||
mkdir stella && cd stella
|
||||
|
||||
# 2. Download the signed Compose bundle + example .env
|
||||
curl -LO https://get.stella-ops.org/releases/latest/.env.example
|
||||
curl -LO https://get.stella-ops.org/releases/latest/.env.example.sig
|
||||
curl -LO https://get.stella-ops.org/releases/latest/docker-compose.infrastructure.yml
|
||||
curl -LO https://get.stella-ops.org/releases/latest/docker-compose.infrastructure.yml.sig
|
||||
curl -LO https://get.stella-ops.org/releases/latest/docker-compose.stella-ops.yml
|
||||
curl -LO https://get.stella-ops.org/releases/latest/docker-compose.stella-ops.yml.sig
|
||||
|
||||
# 3. Verify provenance (Cosign public key is stable)
|
||||
cosign verify-blob \
|
||||
--key https://stella-ops.org/keys/cosign.pub \
|
||||
--signature .env.example.sig \
|
||||
.env.example
|
||||
|
||||
cosign verify-blob \
|
||||
--key https://stella-ops.org/keys/cosign.pub \
|
||||
--signature docker-compose.infrastructure.yml.sig \
|
||||
docker-compose.infrastructure.yml
|
||||
|
||||
cosign verify-blob \
|
||||
--key https://stella-ops.org/keys/cosign.pub \
|
||||
--signature docker-compose.stella-ops.yml.sig \
|
||||
docker-compose.stella-ops.yml
|
||||
|
||||
# 4. Copy .env.example → .env and edit secrets
|
||||
cp .env.example .env
|
||||
$EDITOR .env
|
||||
|
||||
# 5. Launch databases (MongoDB + Redis)
|
||||
docker compose --env-file .env -f docker-compose.infrastructure.yml up -d
|
||||
|
||||
# 6. Launch Stella Ops (first run pulls ~50 MB merged vuln DB)
|
||||
docker compose --env-file .env -f docker-compose.stella-ops.yml up -d
|
||||
````
|
||||
|
||||
*Default login:* `admin / changeme`
|
||||
UI: [https://\<host\>:8443](https://<host>:8443) (self‑signed certificate)
|
||||
|
||||
> **Pinning best‑practice** – in production environments replace
|
||||
> `stella-ops:latest` with the immutable digest printed by
|
||||
> `docker images --digests`.
|
||||
|
||||
> **Repo bundles** – Development, staging, and air‑gapped Compose profiles live
|
||||
> under `deploy/compose/`, already tied to the release manifests in
|
||||
> `deploy/releases/`. Helm users can pull the same channel overlays from
|
||||
> `deploy/helm/stellaops/values-*.yaml` and validate everything with
|
||||
> `deploy/tools/validate-profiles.sh`.
|
||||
|
||||
### 1.1 · Concelier authority configuration
|
||||
|
||||
The Concelier container reads configuration from `etc/concelier.yaml` plus
|
||||
`CONCELIER_` environment variables. To enable the new Authority integration:
|
||||
|
||||
1. Add the following keys to `.env` (replace values for your environment):
|
||||
|
||||
```bash
|
||||
CONCELIER_AUTHORITY__ENABLED=true
|
||||
CONCELIER_AUTHORITY__ALLOWANONYMOUSFALLBACK=true # temporary rollout only
|
||||
CONCELIER_AUTHORITY__ISSUER="https://authority.internal"
|
||||
CONCELIER_AUTHORITY__AUDIENCES__0="api://concelier"
|
||||
CONCELIER_AUTHORITY__REQUIREDSCOPES__0="concelier.jobs.trigger"
|
||||
CONCELIER_AUTHORITY__REQUIREDSCOPES__1="advisory:read"
|
||||
CONCELIER_AUTHORITY__REQUIREDSCOPES__2="advisory:ingest"
|
||||
CONCELIER_AUTHORITY__REQUIREDTENANTS__0="tenant-default"
|
||||
CONCELIER_AUTHORITY__CLIENTID="concelier-jobs"
|
||||
CONCELIER_AUTHORITY__CLIENTSCOPES__0="concelier.jobs.trigger"
|
||||
CONCELIER_AUTHORITY__CLIENTSCOPES__1="advisory:read"
|
||||
CONCELIER_AUTHORITY__CLIENTSCOPES__2="advisory:ingest"
|
||||
CONCELIER_AUTHORITY__CLIENTSECRETFILE="/run/secrets/concelier_authority_client"
|
||||
CONCELIER_AUTHORITY__BYPASSNETWORKS__0="127.0.0.1/32"
|
||||
CONCELIER_AUTHORITY__BYPASSNETWORKS__1="::1/128"
|
||||
CONCELIER_AUTHORITY__RESILIENCE__ENABLERETRIES=true
|
||||
CONCELIER_AUTHORITY__RESILIENCE__RETRYDELAYS__0="00:00:01"
|
||||
CONCELIER_AUTHORITY__RESILIENCE__RETRYDELAYS__1="00:00:02"
|
||||
CONCELIER_AUTHORITY__RESILIENCE__RETRYDELAYS__2="00:00:05"
|
||||
CONCELIER_AUTHORITY__RESILIENCE__ALLOWOFFLINECACHEFALLBACK=true
|
||||
CONCELIER_AUTHORITY__RESILIENCE__OFFLINECACHETOLERANCE="00:10:00"
|
||||
```
|
||||
|
||||
Store the client secret outside source control (Docker secrets, mounted file,
|
||||
or Kubernetes Secret). Concelier loads the secret during post-configuration, so
|
||||
the value never needs to appear in the YAML template.
|
||||
|
||||
Connected sites can keep the retry ladder short (1 s, 2 s, 5 s) so job triggers fail fast when Authority is down. For air‑gapped or intermittently connected deployments, extend `RESILIENCE__OFFLINECACHETOLERANCE` (e.g. `00:30:00`) so cached discovery/JWKS data remains valid while the Offline Kit synchronises upstream changes.
|
||||
|
||||
2. Redeploy Concelier:
|
||||
|
||||
```bash
|
||||
docker compose --env-file .env -f docker-compose.stella-ops.yml up -d concelier
|
||||
```
|
||||
|
||||
3. Tail the logs: `docker compose logs -f concelier`. Successful `/jobs*` calls now
|
||||
emit `Concelier.Authorization.Audit` entries with `route`, `status`, `subject`,
|
||||
`clientId`, `scopes`, `bypass`, and `remote` fields. 401 denials keep the same
|
||||
shape—watch for `bypass=True`, which indicates a bypass CIDR accepted an anonymous
|
||||
call. See `docs/ops/concelier-authority-audit-runbook.md` for a full audit/alerting checklist.
|
||||
|
||||
> **Enforcement deadline** – keep `CONCELIER_AUTHORITY__ALLOWANONYMOUSFALLBACK=true`
|
||||
> only while validating the rollout. Set it to `false` (and restart Concelier)
|
||||
> before **2025-12-31 UTC** to require tokens in production.
|
||||
|
||||
---
|
||||
|
||||
## 2 · Optional: request a free quota token
|
||||
|
||||
Anonymous installs allow **{{ quota\_anon }} scans per UTC day**.
|
||||
Email `token@stella-ops.org` to receive a signed JWT that raises the limit to
|
||||
**{{ quota\_token }} scans/day**. Insert it into `.env`:
|
||||
|
||||
```bash
|
||||
STELLA_JWT="paste‑token‑here"
|
||||
docker compose --env-file .env -f docker-compose.stella-ops.yml \
|
||||
exec stella-ops stella set-jwt "$STELLA_JWT"
|
||||
```
|
||||
|
||||
> The UI shows a reminder at 200 scans and throttles above the limit but will
|
||||
> **never block** your pipeline.
|
||||
|
||||
---
|
||||
|
||||
## 3 · Air‑gapped install (Offline Update Kit)
|
||||
|
||||
When running on an isolated network use the **Offline Update Kit (OUK)**:
|
||||
|
||||
```bash
|
||||
# Download & verify on a connected host
|
||||
curl -LO https://get.stella-ops.org/ouk/stella-ops-offline-kit-v0.1a.tgz
|
||||
curl -LO https://get.stella-ops.org/ouk/stella-ops-offline-kit-v0.1a.tgz.sig
|
||||
|
||||
cosign verify-blob \
|
||||
--key https://stella-ops.org/keys/cosign.pub \
|
||||
--signature stella-ops-offline-kit-v0.1a.tgz.sig \
|
||||
stella-ops-offline-kit-v0.1a.tgz
|
||||
|
||||
# Transfer → air‑gap → import
|
||||
docker compose --env-file .env -f docker-compose.stella-ops.yml \
|
||||
exec stella admin import-offline-usage-kit stella-ops-offline-kit-v0.1a.tgz
|
||||
```
|
||||
|
||||
*Import is atomic; no service downtime.*
|
||||
|
||||
For details see the dedicated [Offline Kit guide](/offline/).
|
||||
|
||||
---
|
||||
|
||||
## 4 · Next steps
|
||||
|
||||
* **5‑min Quick‑Start:** `/quickstart/`
|
||||
* **CI recipes:** `docs/ci/20_CI_RECIPES.md`
|
||||
* **Plug‑in SDK:** `/plugins/`
|
||||
|
||||
---
|
||||
|
||||
*Generated {{ "now" | date: "%Y‑%m‑%d" }} — build tags inserted at render time.*
|
||||
# Stella Ops — Installation Guide (Docker & Air‑Gap)
|
||||
|
||||
<!--
|
||||
This file is processed by the Eleventy build.
|
||||
Do **not** hard‑code versions or quota numbers; inherit from
|
||||
docs/_includes/CONSTANTS.md instead.
|
||||
{{ dotnet }} → ".NET 10 LTS"
|
||||
{{ angular }} → "20"
|
||||
-->
|
||||
|
||||
> **Status — public α not yet published.**
|
||||
> The commands below will work as soon as the first image is tagged
|
||||
> `registry.stella-ops.org/stella-ops/stella-ops:0.1.0-alpha`
|
||||
> (target date: **late 2025**). Track progress on the
|
||||
> [road‑map](/roadmap/).
|
||||
|
||||
---
|
||||
|
||||
## 0 · Prerequisites
|
||||
|
||||
| Item | Minimum | Notes |
|
||||
|------|---------|-------|
|
||||
| Linux | Ubuntu 22.04 LTS / Alma 9 | x86‑64 or arm64 |
|
||||
| CPU / RAM | 2 vCPU / 2 GiB | Laptop baseline |
|
||||
| Disk | 10 GiB SSD | SBOM + vuln DB cache |
|
||||
| Docker | **Engine 25 + Compose v2** | `docker -v` |
|
||||
| TLS | OpenSSL 1.1 + | Self‑signed cert generated at first run |
|
||||
|
||||
---
|
||||
|
||||
## 1 · Connected‑host install (Docker Compose)
|
||||
|
||||
```bash
|
||||
# 1. Make a working directory
|
||||
mkdir stella && cd stella
|
||||
|
||||
# 2. Download the signed Compose bundle + example .env
|
||||
curl -LO https://get.stella-ops.org/releases/latest/.env.example
|
||||
curl -LO https://get.stella-ops.org/releases/latest/.env.example.sig
|
||||
curl -LO https://get.stella-ops.org/releases/latest/docker-compose.infrastructure.yml
|
||||
curl -LO https://get.stella-ops.org/releases/latest/docker-compose.infrastructure.yml.sig
|
||||
curl -LO https://get.stella-ops.org/releases/latest/docker-compose.stella-ops.yml
|
||||
curl -LO https://get.stella-ops.org/releases/latest/docker-compose.stella-ops.yml.sig
|
||||
|
||||
# 3. Verify provenance (Cosign public key is stable)
|
||||
cosign verify-blob \
|
||||
--key https://stella-ops.org/keys/cosign.pub \
|
||||
--signature .env.example.sig \
|
||||
.env.example
|
||||
|
||||
cosign verify-blob \
|
||||
--key https://stella-ops.org/keys/cosign.pub \
|
||||
--signature docker-compose.infrastructure.yml.sig \
|
||||
docker-compose.infrastructure.yml
|
||||
|
||||
cosign verify-blob \
|
||||
--key https://stella-ops.org/keys/cosign.pub \
|
||||
--signature docker-compose.stella-ops.yml.sig \
|
||||
docker-compose.stella-ops.yml
|
||||
|
||||
# 4. Copy .env.example → .env and edit secrets
|
||||
cp .env.example .env
|
||||
$EDITOR .env
|
||||
|
||||
# 5. Launch databases (MongoDB + Redis)
|
||||
docker compose --env-file .env -f docker-compose.infrastructure.yml up -d
|
||||
|
||||
# 6. Launch Stella Ops (first run pulls ~50 MB merged vuln DB)
|
||||
docker compose --env-file .env -f docker-compose.stella-ops.yml up -d
|
||||
````
|
||||
|
||||
*Default login:* `admin / changeme`
|
||||
UI: [https://\<host\>:8443](https://<host>:8443) (self‑signed certificate)
|
||||
|
||||
> **Pinning best‑practice** – in production environments replace
|
||||
> `stella-ops:latest` with the immutable digest printed by
|
||||
> `docker images --digests`.
|
||||
|
||||
> **Repo bundles** – Development, staging, and air‑gapped Compose profiles live
|
||||
> under `deploy/compose/`, already tied to the release manifests in
|
||||
> `deploy/releases/`. Helm users can pull the same channel overlays from
|
||||
> `deploy/helm/stellaops/values-*.yaml` and validate everything with
|
||||
> `deploy/tools/validate-profiles.sh`.
|
||||
|
||||
### 1.1 · Concelier authority configuration
|
||||
|
||||
The Concelier container reads configuration from `etc/concelier.yaml` plus
|
||||
`CONCELIER_` environment variables. To enable the new Authority integration:
|
||||
|
||||
1. Add the following keys to `.env` (replace values for your environment):
|
||||
|
||||
```bash
|
||||
CONCELIER_AUTHORITY__ENABLED=true
|
||||
CONCELIER_AUTHORITY__ALLOWANONYMOUSFALLBACK=true # temporary rollout only
|
||||
CONCELIER_AUTHORITY__ISSUER="https://authority.internal"
|
||||
CONCELIER_AUTHORITY__AUDIENCES__0="api://concelier"
|
||||
CONCELIER_AUTHORITY__REQUIREDSCOPES__0="concelier.jobs.trigger"
|
||||
CONCELIER_AUTHORITY__REQUIREDSCOPES__1="advisory:read"
|
||||
CONCELIER_AUTHORITY__REQUIREDSCOPES__2="advisory:ingest"
|
||||
CONCELIER_AUTHORITY__REQUIREDTENANTS__0="tenant-default"
|
||||
CONCELIER_AUTHORITY__CLIENTID="concelier-jobs"
|
||||
CONCELIER_AUTHORITY__CLIENTSCOPES__0="concelier.jobs.trigger"
|
||||
CONCELIER_AUTHORITY__CLIENTSCOPES__1="advisory:read"
|
||||
CONCELIER_AUTHORITY__CLIENTSCOPES__2="advisory:ingest"
|
||||
CONCELIER_AUTHORITY__CLIENTSECRETFILE="/run/secrets/concelier_authority_client"
|
||||
CONCELIER_AUTHORITY__BYPASSNETWORKS__0="127.0.0.1/32"
|
||||
CONCELIER_AUTHORITY__BYPASSNETWORKS__1="::1/128"
|
||||
CONCELIER_AUTHORITY__RESILIENCE__ENABLERETRIES=true
|
||||
CONCELIER_AUTHORITY__RESILIENCE__RETRYDELAYS__0="00:00:01"
|
||||
CONCELIER_AUTHORITY__RESILIENCE__RETRYDELAYS__1="00:00:02"
|
||||
CONCELIER_AUTHORITY__RESILIENCE__RETRYDELAYS__2="00:00:05"
|
||||
CONCELIER_AUTHORITY__RESILIENCE__ALLOWOFFLINECACHEFALLBACK=true
|
||||
CONCELIER_AUTHORITY__RESILIENCE__OFFLINECACHETOLERANCE="00:10:00"
|
||||
```
|
||||
|
||||
Store the client secret outside source control (Docker secrets, mounted file,
|
||||
or Kubernetes Secret). Concelier loads the secret during post-configuration, so
|
||||
the value never needs to appear in the YAML template.
|
||||
|
||||
Connected sites can keep the retry ladder short (1 s, 2 s, 5 s) so job triggers fail fast when Authority is down. For air‑gapped or intermittently connected deployments, extend `RESILIENCE__OFFLINECACHETOLERANCE` (e.g. `00:30:00`) so cached discovery/JWKS data remains valid while the Offline Kit synchronises upstream changes.
|
||||
|
||||
2. Redeploy Concelier:
|
||||
|
||||
```bash
|
||||
docker compose --env-file .env -f docker-compose.stella-ops.yml up -d concelier
|
||||
```
|
||||
|
||||
3. Tail the logs: `docker compose logs -f concelier`. Successful `/jobs*` calls now
|
||||
emit `Concelier.Authorization.Audit` entries with `route`, `status`, `subject`,
|
||||
`clientId`, `scopes`, `bypass`, and `remote` fields. 401 denials keep the same
|
||||
shape—watch for `bypass=True`, which indicates a bypass CIDR accepted an anonymous
|
||||
call. See `docs/ops/concelier-authority-audit-runbook.md` for a full audit/alerting checklist.
|
||||
|
||||
> **Enforcement deadline** – keep `CONCELIER_AUTHORITY__ALLOWANONYMOUSFALLBACK=true`
|
||||
> only while validating the rollout. Set it to `false` (and restart Concelier)
|
||||
> before **2025-12-31 UTC** to require tokens in production.
|
||||
|
||||
---
|
||||
|
||||
## 2 · Optional: request a free quota token
|
||||
|
||||
Anonymous installs allow **{{ quota\_anon }} scans per UTC day**.
|
||||
Email `token@stella-ops.org` to receive a signed JWT that raises the limit to
|
||||
**{{ quota\_token }} scans/day**. Insert it into `.env`:
|
||||
|
||||
```bash
|
||||
STELLA_JWT="paste‑token‑here"
|
||||
docker compose --env-file .env -f docker-compose.stella-ops.yml \
|
||||
exec stella-ops stella set-jwt "$STELLA_JWT"
|
||||
```
|
||||
|
||||
> The UI shows a reminder at 200 scans and throttles above the limit but will
|
||||
> **never block** your pipeline.
|
||||
|
||||
---
|
||||
|
||||
## 3 · Air‑gapped install (Offline Update Kit)
|
||||
|
||||
When running on an isolated network use the **Offline Update Kit (OUK)**:
|
||||
|
||||
```bash
|
||||
# Download & verify on a connected host
|
||||
curl -LO https://get.stella-ops.org/ouk/stella-ops-offline-kit-v0.1a.tgz
|
||||
curl -LO https://get.stella-ops.org/ouk/stella-ops-offline-kit-v0.1a.tgz.sig
|
||||
|
||||
cosign verify-blob \
|
||||
--key https://stella-ops.org/keys/cosign.pub \
|
||||
--signature stella-ops-offline-kit-v0.1a.tgz.sig \
|
||||
stella-ops-offline-kit-v0.1a.tgz
|
||||
|
||||
# Transfer → air‑gap → import
|
||||
docker compose --env-file .env -f docker-compose.stella-ops.yml \
|
||||
exec stella admin import-offline-usage-kit stella-ops-offline-kit-v0.1a.tgz
|
||||
```
|
||||
|
||||
*Import is atomic; no service downtime.*
|
||||
|
||||
For details see the dedicated [Offline Kit guide](/offline/).
|
||||
|
||||
---
|
||||
|
||||
## 4 · Next steps
|
||||
|
||||
* **5‑min Quick‑Start:** `/quickstart/`
|
||||
* **CI recipes:** `docs/ci/20_CI_RECIPES.md`
|
||||
* **Plug‑in SDK:** `/plugins/`
|
||||
|
||||
---
|
||||
|
||||
*Generated {{ "now" | date: "%Y‑%m‑%d" }} — build tags inserted at render time.*
|
||||
|
||||
@@ -1,443 +1,443 @@
|
||||
# component_architecture_authority.md — **Stella Ops Authority** (2025Q4)
|
||||
|
||||
> **Scope.** Implementation‑ready architecture for **Stella Ops Authority**: the on‑prem **OIDC/OAuth2** service that issues **short‑lived, sender‑constrained operational tokens (OpToks)** to first‑party services and tools. Covers protocols (DPoP & mTLS binding), token shapes, endpoints, storage, rotation, HA, RBAC, audit, and testing. This component is the trust anchor for *who* is calling inside a Stella Ops installation. (Entitlement is proven separately by **PoE** from the cloud Licensing Service; Authority does not issue PoE.)
|
||||
|
||||
---
|
||||
|
||||
## 0) Mission & boundaries
|
||||
|
||||
**Mission.** Provide **fast, local, verifiable** authentication for Stella Ops microservices and tools by minting **very short‑lived** OAuth2/OIDC tokens that are **sender‑constrained** (DPoP or mTLS‑bound). Support RBAC scopes, multi‑tenant claims, and deterministic validation for APIs (Scanner, Signer, Attestor, Excititor, Concelier, UI, CLI, Zastava).
|
||||
|
||||
**Boundaries.**
|
||||
|
||||
* Authority **does not** validate entitlements/licensing. That’s enforced by **Signer** using **PoE** with the cloud Licensing Service.
|
||||
* Authority tokens are **operational only** (2–5 min TTL) and must not be embedded in long‑lived artifacts or stored in SBOMs.
|
||||
* Authority is **stateless for validation** (JWT) and **optional introspection** for services that prefer online checks.
|
||||
|
||||
---
|
||||
|
||||
## 1) Protocols & cryptography
|
||||
|
||||
* **OIDC Discovery**: `/.well-known/openid-configuration`
|
||||
* **OAuth2** grant types:
|
||||
|
||||
* **Client Credentials** (service↔service, with mTLS or private_key_jwt)
|
||||
* **Device Code** (CLI login on headless agents; optional)
|
||||
* **Authorization Code + PKCE** (browser login for UI; optional)
|
||||
* **Sender constraint options** (choose per caller or per audience):
|
||||
|
||||
* **DPoP** (Demonstration of Proof‑of‑Possession): proof JWT on each HTTP request, bound to the access token via `cnf.jkt`.
|
||||
* **OAuth 2.0 mTLS** (certificate‑bound tokens): token bound to client certificate thumbprint via `cnf.x5t#S256`.
|
||||
* **Signing algorithms**: **EdDSA (Ed25519)** preferred; fallback **ES256 (P‑256)**. Rotation is supported via **kid** in JWKS.
|
||||
* **Token format**: **JWT** access tokens (compact), optionally opaque reference tokens for services that insist on introspection.
|
||||
* **Clock skew tolerance**: ±60 s; issue `nbf`, `iat`, `exp` accordingly.
|
||||
|
||||
---
|
||||
|
||||
## 2) Token model
|
||||
|
||||
### 2.1 Access token (OpTok) — short‑lived (120–300 s)
|
||||
|
||||
**Registered claims**
|
||||
|
||||
```
|
||||
iss = https://authority.<domain>
|
||||
sub = <client_id or user_id>
|
||||
aud = <service audience: signer|scanner|attestor|concelier|excititor|ui|zastava>
|
||||
exp = <unix ts> (<= 300 s from iat)
|
||||
iat = <unix ts>
|
||||
nbf = iat - 30
|
||||
jti = <uuid>
|
||||
scope = "scanner.scan scanner.export signer.sign ..."
|
||||
```
|
||||
|
||||
**Sender‑constraint (`cnf`)**
|
||||
|
||||
* **DPoP**:
|
||||
|
||||
```json
|
||||
"cnf": { "jkt": "<base64url(SHA-256(JWK))>" }
|
||||
```
|
||||
* **mTLS**:
|
||||
|
||||
```json
|
||||
"cnf": { "x5t#S256": "<base64url(SHA-256(client_cert_der))>" }
|
||||
```
|
||||
|
||||
**Install/tenant context (custom claims)**
|
||||
|
||||
```
|
||||
tid = <tenant id> // multi-tenant
|
||||
inst = <installation id> // unique installation
|
||||
roles = [ "svc.scanner", "svc.signer", "ui.admin", ... ]
|
||||
plan? = <plan name> // optional hint for UIs; not used for enforcement
|
||||
```
|
||||
|
||||
> **Note**: Do **not** copy PoE claims into OpTok; OpTok ≠ entitlement. Only **Signer** checks PoE.
|
||||
|
||||
### 2.2 Refresh tokens (optional)
|
||||
|
||||
* Default **disabled**. If enabled (for UI interactive logins), pair with **DPoP‑bound** refresh tokens or **mTLS** client sessions; short TTL (≤ 8 h), rotating on use (replay‑safe).
|
||||
|
||||
### 2.3 ID tokens (optional)
|
||||
|
||||
* Issued for UI/browser OIDC flows (Authorization Code + PKCE); not used for service auth.
|
||||
|
||||
---
|
||||
|
||||
## 3) Endpoints & flows
|
||||
|
||||
### 3.1 OIDC discovery & keys
|
||||
|
||||
* `GET /.well-known/openid-configuration` → endpoints, algs, jwks_uri
|
||||
* `GET /jwks` → JSON Web Key Set (rotating, at least 2 active keys during transition)
|
||||
|
||||
### 3.2 Token issuance
|
||||
|
||||
* `POST /oauth/token`
|
||||
|
||||
* **Client Credentials** (service→service):
|
||||
|
||||
# component_architecture_authority.md — **Stella Ops Authority** (2025Q4)
|
||||
|
||||
> **Scope.** Implementation‑ready architecture for **Stella Ops Authority**: the on‑prem **OIDC/OAuth2** service that issues **short‑lived, sender‑constrained operational tokens (OpToks)** to first‑party services and tools. Covers protocols (DPoP & mTLS binding), token shapes, endpoints, storage, rotation, HA, RBAC, audit, and testing. This component is the trust anchor for *who* is calling inside a Stella Ops installation. (Entitlement is proven separately by **PoE** from the cloud Licensing Service; Authority does not issue PoE.)
|
||||
|
||||
---
|
||||
|
||||
## 0) Mission & boundaries
|
||||
|
||||
**Mission.** Provide **fast, local, verifiable** authentication for Stella Ops microservices and tools by minting **very short‑lived** OAuth2/OIDC tokens that are **sender‑constrained** (DPoP or mTLS‑bound). Support RBAC scopes, multi‑tenant claims, and deterministic validation for APIs (Scanner, Signer, Attestor, Excititor, Concelier, UI, CLI, Zastava).
|
||||
|
||||
**Boundaries.**
|
||||
|
||||
* Authority **does not** validate entitlements/licensing. That’s enforced by **Signer** using **PoE** with the cloud Licensing Service.
|
||||
* Authority tokens are **operational only** (2–5 min TTL) and must not be embedded in long‑lived artifacts or stored in SBOMs.
|
||||
* Authority is **stateless for validation** (JWT) and **optional introspection** for services that prefer online checks.
|
||||
|
||||
---
|
||||
|
||||
## 1) Protocols & cryptography
|
||||
|
||||
* **OIDC Discovery**: `/.well-known/openid-configuration`
|
||||
* **OAuth2** grant types:
|
||||
|
||||
* **Client Credentials** (service↔service, with mTLS or private_key_jwt)
|
||||
* **Device Code** (CLI login on headless agents; optional)
|
||||
* **Authorization Code + PKCE** (browser login for UI; optional)
|
||||
* **Sender constraint options** (choose per caller or per audience):
|
||||
|
||||
* **DPoP** (Demonstration of Proof‑of‑Possession): proof JWT on each HTTP request, bound to the access token via `cnf.jkt`.
|
||||
* **OAuth 2.0 mTLS** (certificate‑bound tokens): token bound to client certificate thumbprint via `cnf.x5t#S256`.
|
||||
* **Signing algorithms**: **EdDSA (Ed25519)** preferred; fallback **ES256 (P‑256)**. Rotation is supported via **kid** in JWKS.
|
||||
* **Token format**: **JWT** access tokens (compact), optionally opaque reference tokens for services that insist on introspection.
|
||||
* **Clock skew tolerance**: ±60 s; issue `nbf`, `iat`, `exp` accordingly.
|
||||
|
||||
---
|
||||
|
||||
## 2) Token model
|
||||
|
||||
### 2.1 Access token (OpTok) — short‑lived (120–300 s)
|
||||
|
||||
**Registered claims**
|
||||
|
||||
```
|
||||
iss = https://authority.<domain>
|
||||
sub = <client_id or user_id>
|
||||
aud = <service audience: signer|scanner|attestor|concelier|excititor|ui|zastava>
|
||||
exp = <unix ts> (<= 300 s from iat)
|
||||
iat = <unix ts>
|
||||
nbf = iat - 30
|
||||
jti = <uuid>
|
||||
scope = "scanner.scan scanner.export signer.sign ..."
|
||||
```
|
||||
|
||||
**Sender‑constraint (`cnf`)**
|
||||
|
||||
* **DPoP**:
|
||||
|
||||
```json
|
||||
"cnf": { "jkt": "<base64url(SHA-256(JWK))>" }
|
||||
```
|
||||
* **mTLS**:
|
||||
|
||||
```json
|
||||
"cnf": { "x5t#S256": "<base64url(SHA-256(client_cert_der))>" }
|
||||
```
|
||||
|
||||
**Install/tenant context (custom claims)**
|
||||
|
||||
```
|
||||
tid = <tenant id> // multi-tenant
|
||||
inst = <installation id> // unique installation
|
||||
roles = [ "svc.scanner", "svc.signer", "ui.admin", ... ]
|
||||
plan? = <plan name> // optional hint for UIs; not used for enforcement
|
||||
```
|
||||
|
||||
> **Note**: Do **not** copy PoE claims into OpTok; OpTok ≠ entitlement. Only **Signer** checks PoE.
|
||||
|
||||
### 2.2 Refresh tokens (optional)
|
||||
|
||||
* Default **disabled**. If enabled (for UI interactive logins), pair with **DPoP‑bound** refresh tokens or **mTLS** client sessions; short TTL (≤ 8 h), rotating on use (replay‑safe).
|
||||
|
||||
### 2.3 ID tokens (optional)
|
||||
|
||||
* Issued for UI/browser OIDC flows (Authorization Code + PKCE); not used for service auth.
|
||||
|
||||
---
|
||||
|
||||
## 3) Endpoints & flows
|
||||
|
||||
### 3.1 OIDC discovery & keys
|
||||
|
||||
* `GET /.well-known/openid-configuration` → endpoints, algs, jwks_uri
|
||||
* `GET /jwks` → JSON Web Key Set (rotating, at least 2 active keys during transition)
|
||||
|
||||
### 3.2 Token issuance
|
||||
|
||||
* `POST /oauth/token`
|
||||
|
||||
* **Client Credentials** (service→service):
|
||||
|
||||
* **mTLS**: mutual TLS + `client_id` → bound token (`cnf.x5t#S256`)
|
||||
* `security.senderConstraints.mtls.enforceForAudiences` forces the mTLS path when requested `aud`/`resource` values intersect high-value audiences (defaults include `signer`). Authority rejects clients attempting to use DPoP/basic secrets for these audiences.
|
||||
* Stored `certificateBindings` are authoritative: thumbprint, subject, issuer, serial number, and SAN values are matched against the presented certificate, with rotation grace applied to activation windows. Failures surface deterministic error codes (e.g. `certificate_binding_subject_mismatch`).
|
||||
* **private_key_jwt**: JWT‑based client auth + **DPoP** header (preferred for tools and CLI)
|
||||
* **Device Code** (CLI): `POST /oauth/device/code` + `POST /oauth/token` poll
|
||||
* **Authorization Code + PKCE** (UI): standard
|
||||
|
||||
**DPoP handshake (example)**
|
||||
|
||||
1. Client prepares **JWK** (ephemeral keypair).
|
||||
2. Client sends **DPoP proof** header with fields:
|
||||
|
||||
```
|
||||
htm=POST
|
||||
htu=https://authority.../oauth/token
|
||||
iat=<now>
|
||||
jti=<uuid>
|
||||
```
|
||||
|
||||
signed with the DPoP private key; header carries JWK.
|
||||
3. Authority validates proof; issues access token with `cnf.jkt=<thumbprint(JWK)>`.
|
||||
4. Client uses the same DPoP key to sign **every subsequent API request** to services (Signer, Scanner, …).
|
||||
|
||||
**mTLS flow**
|
||||
|
||||
* Mutual TLS at the connection; Authority extracts client cert, validates chain; token carries `cnf.x5t#S256`.
|
||||
|
||||
### 3.3 Introspection & revocation (optional)
|
||||
|
||||
* `POST /oauth/introspect` → `{ active, sub, scope, aud, exp, cnf, ... }`
|
||||
* `POST /oauth/revoke` → revokes refresh tokens or opaque access tokens.
|
||||
* **Replay prevention**: maintain **DPoP `jti` cache** (TTL ≤ 10 min) to reject duplicate proofs when services supply DPoP nonces (Signer requires nonce for high‑value operations).
|
||||
|
||||
### 3.4 UserInfo (optional for UI)
|
||||
|
||||
* `GET /userinfo` (ID token context).
|
||||
|
||||
---
|
||||
|
||||
## 4) Audiences, scopes & RBAC
|
||||
|
||||
### 4.1 Audiences
|
||||
|
||||
* `signer` — only the **Signer** service should accept tokens with `aud=signer`.
|
||||
* `attestor`, `scanner`, `concelier`, `excititor`, `ui`, `zastava` similarly.
|
||||
|
||||
Services **must** verify `aud` and **sender constraint** (DPoP/mTLS) per their policy.
|
||||
|
||||
### 4.2 Core scopes
|
||||
|
||||
| Scope | Service | Operation |
|
||||
| ---------------------------------- | ------------------ | -------------------------- |
|
||||
| `signer.sign` | Signer | Request DSSE signing |
|
||||
| `attestor.write` | Attestor | Submit Rekor entries |
|
||||
| `scanner.scan` | Scanner.WebService | Submit scan jobs |
|
||||
| `scanner.export` | Scanner.WebService | Export SBOMs |
|
||||
| `scanner.read` | Scanner.WebService | Read catalog/SBOMs |
|
||||
| `vex.read` / `vex.admin` | Excititor | Query/operate |
|
||||
| `concelier.read` / `concelier.export` | Concelier | Query/exports |
|
||||
| `ui.read` / `ui.admin` | UI | View/admin |
|
||||
| `zastava.emit` / `zastava.enforce` | Scanner/Zastava | Runtime events / admission |
|
||||
|
||||
**Roles → scopes mapping** is configured centrally (Authority policy) and pushed during token issuance.
|
||||
|
||||
---
|
||||
|
||||
## 5) Storage & state
|
||||
|
||||
* **Configuration DB** (PostgreSQL/MySQL): clients, audiences, role→scope maps, tenant/installation registry, device code grants, persistent consents (if any).
|
||||
* **Cache** (Redis):
|
||||
|
||||
* DPoP **jti** replay cache (short TTL)
|
||||
* **Nonce** store (per resource server, if they demand nonce)
|
||||
* Device code pollers, rate limiting buckets
|
||||
* **JWKS**: key material in HSM/KMS or encrypted at rest; JWKS served from memory.
|
||||
|
||||
---
|
||||
|
||||
## 6) Key management & rotation
|
||||
|
||||
* Maintain **at least 2 signing keys** active during rotation; tokens carry `kid`.
|
||||
* Prefer **Ed25519** for compact tokens; maintain **ES256** fallback for FIPS contexts.
|
||||
* Rotation cadence: 30–90 days; emergency rotation supported.
|
||||
* Publish new JWKS **before** issuing tokens with the new `kid` to avoid cold‑start validation misses.
|
||||
* Keep **old keys** available **at least** for max token TTL + 5 minutes.
|
||||
|
||||
---
|
||||
|
||||
## 7) HA & performance
|
||||
|
||||
* **Stateless issuance** (except device codes/refresh) → scale horizontally behind a load‑balancer.
|
||||
* **DB** only for client metadata and optional flows; token checks are JWT‑local; introspection endpoints hit cache/DB minimally.
|
||||
* **Targets**:
|
||||
|
||||
* Token issuance P95 ≤ **20 ms** under warm cache.
|
||||
* DPoP proof validation ≤ **1 ms** extra per request at resource servers (Signer/Scanner).
|
||||
* 99.9% uptime; HPA on CPU/latency.
|
||||
|
||||
---
|
||||
|
||||
## 8) Security posture
|
||||
|
||||
* **Strict TLS** (1.3 preferred); HSTS; modern cipher suites.
|
||||
* **mTLS** enabled where required (Signer/Attestor paths).
|
||||
* **Replay protection**: DPoP `jti` cache, nonce support for **Signer** (add `DPoP-Nonce` header on 401; clients re‑sign).
|
||||
* **Rate limits** per client & per IP; exponential backoff on failures.
|
||||
* **Secrets**: clients use **private_key_jwt** or **mTLS**; never basic secrets over the wire.
|
||||
* **CSP/CSRF** hardening on UI flows; `SameSite=Lax` cookies; PKCE enforced.
|
||||
* **Logs** redact `Authorization` and DPoP proofs; store `sub`, `aud`, `scopes`, `inst`, `tid`, `cnf` thumbprints, not full keys.
|
||||
|
||||
---
|
||||
|
||||
## 9) Multi‑tenancy & installations
|
||||
|
||||
* **Tenant (`tid`)** and **Installation (`inst`)** registries define which audiences/scopes a client can request.
|
||||
* Cross‑tenant isolation enforced at issuance (disallow rogue `aud`), and resource servers **must** check that `tid` matches their configured tenant.
|
||||
|
||||
---
|
||||
|
||||
## 10) Admin & operations APIs
|
||||
|
||||
All under `/admin` (mTLS + `authority.admin` scope).
|
||||
|
||||
```
|
||||
POST /admin/clients # create/update client (confidential/public)
|
||||
POST /admin/audiences # register audience resource URIs
|
||||
POST /admin/roles # define role→scope mappings
|
||||
POST /admin/tenants # create tenant/install entries
|
||||
POST /admin/keys/rotate # rotate signing key (zero-downtime)
|
||||
GET /admin/metrics # Prometheus exposition (token issue rates, errors)
|
||||
GET /admin/healthz|readyz # health/readiness
|
||||
```
|
||||
|
||||
Declared client `audiences` flow through to the issued JWT `aud` claim and the token request's `resource` indicators. Authority relies on this metadata to enforce DPoP nonce challenges for `signer`, `attestor`, and other high-value services without requiring clients to repeat the audience parameter on every request.
|
||||
|
||||
---
|
||||
|
||||
## 11) Integration hard lines (what resource servers must enforce)
|
||||
|
||||
Every Stella Ops service that consumes Authority tokens **must**:
|
||||
|
||||
1. Verify JWT signature (`kid` in JWKS), `iss`, `aud`, `exp`, `nbf`.
|
||||
2. Enforce **sender‑constraint**:
|
||||
|
||||
* **DPoP**: validate DPoP proof (`htu`, `htm`, `iat`, `jti`) and match `cnf.jkt`; cache `jti` for replay defense; honor nonce challenges.
|
||||
* **mTLS**: match presented client cert thumbprint to token `cnf.x5t#S256`.
|
||||
3. Check **scopes**; optionally map to internal roles.
|
||||
4. Check **tenant** (`tid`) and **installation** (`inst`) as appropriate.
|
||||
5. For **Signer** only: require **both** OpTok and **PoE** in the request (enforced by Signer, not Authority).
|
||||
|
||||
---
|
||||
|
||||
## 12) Error surfaces & UX
|
||||
|
||||
* Token endpoint errors follow OAuth2 (`invalid_client`, `invalid_grant`, `invalid_scope`, `unauthorized_client`).
|
||||
* Resource servers use RFC 6750 style (`WWW-Authenticate: DPoP error="invalid_token", error_description="…", dpop_nonce="…" `).
|
||||
* For DPoP nonce challenges, clients retry with the server‑supplied nonce once.
|
||||
|
||||
---
|
||||
|
||||
## 13) Observability & audit
|
||||
|
||||
* **Metrics**:
|
||||
|
||||
* `authority.tokens_issued_total{grant,aud}`
|
||||
* `authority.dpop_validations_total{result}`
|
||||
* `authority.mtls_bindings_total{result}`
|
||||
* `authority.jwks_rotations_total`
|
||||
* `authority.errors_total{type}`
|
||||
* **Audit log** (immutable sink): token issuance (`sub`, `aud`, `scopes`, `tid`, `inst`, `cnf thumbprint`, `jti`), revocations, admin changes.
|
||||
* **Tracing**: token flows, DB reads, JWKS cache.
|
||||
|
||||
---
|
||||
|
||||
## 14) Configuration (YAML)
|
||||
|
||||
```yaml
|
||||
authority:
|
||||
issuer: "https://authority.internal"
|
||||
signing:
|
||||
enabled: true
|
||||
activeKeyId: "authority-signing-2025"
|
||||
keyPath: "../certificates/authority-signing-2025.pem"
|
||||
algorithm: "ES256"
|
||||
keySource: "file"
|
||||
security:
|
||||
rateLimiting:
|
||||
token:
|
||||
enabled: true
|
||||
permitLimit: 30
|
||||
window: "00:01:00"
|
||||
queueLimit: 0
|
||||
authorize:
|
||||
enabled: true
|
||||
permitLimit: 60
|
||||
window: "00:01:00"
|
||||
queueLimit: 10
|
||||
internal:
|
||||
enabled: false
|
||||
permitLimit: 5
|
||||
window: "00:01:00"
|
||||
queueLimit: 0
|
||||
senderConstraints:
|
||||
dpop:
|
||||
enabled: true
|
||||
allowedAlgorithms: [ "ES256", "ES384" ]
|
||||
proofLifetime: "00:02:00"
|
||||
allowedClockSkew: "00:00:30"
|
||||
replayWindow: "00:05:00"
|
||||
nonce:
|
||||
enabled: true
|
||||
ttl: "00:10:00"
|
||||
maxIssuancePerMinute: 120
|
||||
store: "redis"
|
||||
redisConnectionString: "redis://authority-redis:6379?ssl=false"
|
||||
requiredAudiences:
|
||||
- "signer"
|
||||
- "attestor"
|
||||
mtls:
|
||||
enabled: true
|
||||
requireChainValidation: true
|
||||
rotationGrace: "00:15:00"
|
||||
enforceForAudiences:
|
||||
- "signer"
|
||||
allowedSanTypes:
|
||||
- "dns"
|
||||
- "uri"
|
||||
allowedCertificateAuthorities:
|
||||
- "/etc/ssl/mtls/clients-ca.pem"
|
||||
clients:
|
||||
- clientId: scanner-web
|
||||
grantTypes: [ "client_credentials" ]
|
||||
audiences: [ "scanner" ]
|
||||
auth: { type: "private_key_jwt", jwkFile: "/secrets/scanner-web.jwk" }
|
||||
senderConstraint: "dpop"
|
||||
scopes: [ "scanner.scan", "scanner.export", "scanner.read" ]
|
||||
- clientId: signer
|
||||
grantTypes: [ "client_credentials" ]
|
||||
audiences: [ "signer" ]
|
||||
auth: { type: "mtls" }
|
||||
senderConstraint: "mtls"
|
||||
scopes: [ "signer.sign" ]
|
||||
- clientId: notify-web-dev
|
||||
grantTypes: [ "client_credentials" ]
|
||||
audiences: [ "notify.dev" ]
|
||||
auth: { type: "client_secret", secretFile: "/secrets/notify-web-dev.secret" }
|
||||
senderConstraint: "dpop"
|
||||
scopes: [ "notify.read", "notify.admin" ]
|
||||
- clientId: notify-web
|
||||
grantTypes: [ "client_credentials" ]
|
||||
audiences: [ "notify" ]
|
||||
auth: { type: "client_secret", secretFile: "/secrets/notify-web.secret" }
|
||||
senderConstraint: "dpop"
|
||||
scopes: [ "notify.read", "notify.admin" ]
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 15) Testing matrix
|
||||
|
||||
* **JWT validation**: wrong `aud`, expired `exp`, skewed `nbf`, stale `kid`.
|
||||
* **DPoP**: invalid `htu`/`htm`, replayed `jti`, stale `iat`, wrong `jkt`, nonce dance.
|
||||
* **mTLS**: wrong client cert, wrong CA, thumbprint mismatch.
|
||||
* **RBAC**: scope enforcement per audience; over‑privileged client denied.
|
||||
* **Rotation**: JWKS rotation while load‑testing; zero‑downtime verification.
|
||||
* **HA**: kill one Authority instance; verify issuance continues; JWKS served by peers.
|
||||
* **Performance**: 1k token issuance/sec on 2 cores with Redis enabled for jti caching.
|
||||
|
||||
---
|
||||
|
||||
## 16) Threat model & mitigations (summary)
|
||||
|
||||
| Threat | Vector | Mitigation |
|
||||
| ------------------- | ---------------- | ------------------------------------------------------------------------------------------ |
|
||||
| Token theft | Copy of JWT | **Short TTL**, **sender‑constraint** (DPoP/mTLS); replay blocked by `jti` cache and nonces |
|
||||
| Replay across hosts | Reuse DPoP proof | Enforce `htu`/`htm`, `iat` freshness, `jti` uniqueness; services may require **nonce** |
|
||||
| Impersonation | Fake client | mTLS or `private_key_jwt` with pinned JWK; client registration & rotation |
|
||||
| Key compromise | Signing key leak | HSM/KMS storage, key rotation, audit; emergency key revoke path; narrow token TTL |
|
||||
| Cross‑tenant abuse | Scope elevation | Enforce `aud`, `tid`, `inst` at issuance and resource servers |
|
||||
| Downgrade to bearer | Strip DPoP | Resource servers require DPoP/mTLS based on `aud`; reject bearer without `cnf` |
|
||||
|
||||
---
|
||||
|
||||
## 17) Deployment & HA
|
||||
|
||||
* **Stateless** microservice, containerized; run ≥ 2 replicas behind LB.
|
||||
* **DB**: HA Postgres (or MySQL) for clients/roles; **Redis** for device codes, DPoP nonces/jtis.
|
||||
* **Secrets**: mount client JWKs via K8s Secrets/HashiCorp Vault; signing keys via KMS.
|
||||
* **Backups**: DB daily; Redis not critical (ephemeral).
|
||||
* **Disaster recovery**: export/import of client registry; JWKS rehydrate from KMS.
|
||||
* **Compliance**: TLS audit; penetration testing for OIDC flows.
|
||||
|
||||
---
|
||||
|
||||
## 18) Implementation notes
|
||||
|
||||
* Reference stack: **.NET 10** + **OpenIddict 6** (or IdentityServer if licensed) with custom DPoP validator and mTLS binding middleware.
|
||||
* Keep the DPoP/JTI cache pluggable; allow Redis/Memcached.
|
||||
* Provide **client SDKs** for C# and Go: DPoP key mgmt, proof generation, nonce handling, token refresh helper.
|
||||
|
||||
---
|
||||
|
||||
## 19) Quick reference — wire examples
|
||||
|
||||
**Access token (payload excerpt)**
|
||||
|
||||
```json
|
||||
{
|
||||
"iss": "https://authority.internal",
|
||||
"sub": "scanner-web",
|
||||
"aud": "signer",
|
||||
"exp": 1760668800,
|
||||
"iat": 1760668620,
|
||||
"nbf": 1760668620,
|
||||
"jti": "9d9c3f01-6e1a-49f1-8f77-9b7e6f7e3c50",
|
||||
"scope": "signer.sign",
|
||||
"tid": "tenant-01",
|
||||
"inst": "install-7A2B",
|
||||
"cnf": { "jkt": "KcVb2V...base64url..." }
|
||||
}
|
||||
```
|
||||
|
||||
**DPoP proof header fields (for POST /sign/dsse)**
|
||||
|
||||
```json
|
||||
{
|
||||
"htu": "https://signer.internal/sign/dsse",
|
||||
"htm": "POST",
|
||||
"iat": 1760668620,
|
||||
"jti": "4b1c9b3c-8a95-4c58-8a92-9c6cfb4a6a0b"
|
||||
}
|
||||
```
|
||||
|
||||
Signer validates that `hash(JWK)` in the proof matches `cnf.jkt` in the token.
|
||||
|
||||
---
|
||||
|
||||
## 20) Rollout plan
|
||||
|
||||
1. **MVP**: Client Credentials (private_key_jwt + DPoP), JWKS, short OpToks, per‑audience scopes.
|
||||
2. **Add**: mTLS‑bound tokens for Signer/Attestor; device code for CLI; optional introspection.
|
||||
3. **Hardening**: DPoP nonce support; full audit pipeline; HA tuning.
|
||||
4. **UX**: Tenant/installation admin UI; role→scope editors; client bootstrap wizards.
|
||||
* **Device Code** (CLI): `POST /oauth/device/code` + `POST /oauth/token` poll
|
||||
* **Authorization Code + PKCE** (UI): standard
|
||||
|
||||
**DPoP handshake (example)**
|
||||
|
||||
1. Client prepares **JWK** (ephemeral keypair).
|
||||
2. Client sends **DPoP proof** header with fields:
|
||||
|
||||
```
|
||||
htm=POST
|
||||
htu=https://authority.../oauth/token
|
||||
iat=<now>
|
||||
jti=<uuid>
|
||||
```
|
||||
|
||||
signed with the DPoP private key; header carries JWK.
|
||||
3. Authority validates proof; issues access token with `cnf.jkt=<thumbprint(JWK)>`.
|
||||
4. Client uses the same DPoP key to sign **every subsequent API request** to services (Signer, Scanner, …).
|
||||
|
||||
**mTLS flow**
|
||||
|
||||
* Mutual TLS at the connection; Authority extracts client cert, validates chain; token carries `cnf.x5t#S256`.
|
||||
|
||||
### 3.3 Introspection & revocation (optional)
|
||||
|
||||
* `POST /oauth/introspect` → `{ active, sub, scope, aud, exp, cnf, ... }`
|
||||
* `POST /oauth/revoke` → revokes refresh tokens or opaque access tokens.
|
||||
* **Replay prevention**: maintain **DPoP `jti` cache** (TTL ≤ 10 min) to reject duplicate proofs when services supply DPoP nonces (Signer requires nonce for high‑value operations).
|
||||
|
||||
### 3.4 UserInfo (optional for UI)
|
||||
|
||||
* `GET /userinfo` (ID token context).
|
||||
|
||||
---
|
||||
|
||||
## 4) Audiences, scopes & RBAC
|
||||
|
||||
### 4.1 Audiences
|
||||
|
||||
* `signer` — only the **Signer** service should accept tokens with `aud=signer`.
|
||||
* `attestor`, `scanner`, `concelier`, `excititor`, `ui`, `zastava` similarly.
|
||||
|
||||
Services **must** verify `aud` and **sender constraint** (DPoP/mTLS) per their policy.
|
||||
|
||||
### 4.2 Core scopes
|
||||
|
||||
| Scope | Service | Operation |
|
||||
| ---------------------------------- | ------------------ | -------------------------- |
|
||||
| `signer.sign` | Signer | Request DSSE signing |
|
||||
| `attestor.write` | Attestor | Submit Rekor entries |
|
||||
| `scanner.scan` | Scanner.WebService | Submit scan jobs |
|
||||
| `scanner.export` | Scanner.WebService | Export SBOMs |
|
||||
| `scanner.read` | Scanner.WebService | Read catalog/SBOMs |
|
||||
| `vex.read` / `vex.admin` | Excititor | Query/operate |
|
||||
| `concelier.read` / `concelier.export` | Concelier | Query/exports |
|
||||
| `ui.read` / `ui.admin` | UI | View/admin |
|
||||
| `zastava.emit` / `zastava.enforce` | Scanner/Zastava | Runtime events / admission |
|
||||
|
||||
**Roles → scopes mapping** is configured centrally (Authority policy) and pushed during token issuance.
|
||||
|
||||
---
|
||||
|
||||
## 5) Storage & state
|
||||
|
||||
* **Configuration DB** (PostgreSQL/MySQL): clients, audiences, role→scope maps, tenant/installation registry, device code grants, persistent consents (if any).
|
||||
* **Cache** (Redis):
|
||||
|
||||
* DPoP **jti** replay cache (short TTL)
|
||||
* **Nonce** store (per resource server, if they demand nonce)
|
||||
* Device code pollers, rate limiting buckets
|
||||
* **JWKS**: key material in HSM/KMS or encrypted at rest; JWKS served from memory.
|
||||
|
||||
---
|
||||
|
||||
## 6) Key management & rotation
|
||||
|
||||
* Maintain **at least 2 signing keys** active during rotation; tokens carry `kid`.
|
||||
* Prefer **Ed25519** for compact tokens; maintain **ES256** fallback for FIPS contexts.
|
||||
* Rotation cadence: 30–90 days; emergency rotation supported.
|
||||
* Publish new JWKS **before** issuing tokens with the new `kid` to avoid cold‑start validation misses.
|
||||
* Keep **old keys** available **at least** for max token TTL + 5 minutes.
|
||||
|
||||
---
|
||||
|
||||
## 7) HA & performance
|
||||
|
||||
* **Stateless issuance** (except device codes/refresh) → scale horizontally behind a load‑balancer.
|
||||
* **DB** only for client metadata and optional flows; token checks are JWT‑local; introspection endpoints hit cache/DB minimally.
|
||||
* **Targets**:
|
||||
|
||||
* Token issuance P95 ≤ **20 ms** under warm cache.
|
||||
* DPoP proof validation ≤ **1 ms** extra per request at resource servers (Signer/Scanner).
|
||||
* 99.9% uptime; HPA on CPU/latency.
|
||||
|
||||
---
|
||||
|
||||
## 8) Security posture
|
||||
|
||||
* **Strict TLS** (1.3 preferred); HSTS; modern cipher suites.
|
||||
* **mTLS** enabled where required (Signer/Attestor paths).
|
||||
* **Replay protection**: DPoP `jti` cache, nonce support for **Signer** (add `DPoP-Nonce` header on 401; clients re‑sign).
|
||||
* **Rate limits** per client & per IP; exponential backoff on failures.
|
||||
* **Secrets**: clients use **private_key_jwt** or **mTLS**; never basic secrets over the wire.
|
||||
* **CSP/CSRF** hardening on UI flows; `SameSite=Lax` cookies; PKCE enforced.
|
||||
* **Logs** redact `Authorization` and DPoP proofs; store `sub`, `aud`, `scopes`, `inst`, `tid`, `cnf` thumbprints, not full keys.
|
||||
|
||||
---
|
||||
|
||||
## 9) Multi‑tenancy & installations
|
||||
|
||||
* **Tenant (`tid`)** and **Installation (`inst`)** registries define which audiences/scopes a client can request.
|
||||
* Cross‑tenant isolation enforced at issuance (disallow rogue `aud`), and resource servers **must** check that `tid` matches their configured tenant.
|
||||
|
||||
---
|
||||
|
||||
## 10) Admin & operations APIs
|
||||
|
||||
All under `/admin` (mTLS + `authority.admin` scope).
|
||||
|
||||
```
|
||||
POST /admin/clients # create/update client (confidential/public)
|
||||
POST /admin/audiences # register audience resource URIs
|
||||
POST /admin/roles # define role→scope mappings
|
||||
POST /admin/tenants # create tenant/install entries
|
||||
POST /admin/keys/rotate # rotate signing key (zero-downtime)
|
||||
GET /admin/metrics # Prometheus exposition (token issue rates, errors)
|
||||
GET /admin/healthz|readyz # health/readiness
|
||||
```
|
||||
|
||||
Declared client `audiences` flow through to the issued JWT `aud` claim and the token request's `resource` indicators. Authority relies on this metadata to enforce DPoP nonce challenges for `signer`, `attestor`, and other high-value services without requiring clients to repeat the audience parameter on every request.
|
||||
|
||||
---
|
||||
|
||||
## 11) Integration hard lines (what resource servers must enforce)
|
||||
|
||||
Every Stella Ops service that consumes Authority tokens **must**:
|
||||
|
||||
1. Verify JWT signature (`kid` in JWKS), `iss`, `aud`, `exp`, `nbf`.
|
||||
2. Enforce **sender‑constraint**:
|
||||
|
||||
* **DPoP**: validate DPoP proof (`htu`, `htm`, `iat`, `jti`) and match `cnf.jkt`; cache `jti` for replay defense; honor nonce challenges.
|
||||
* **mTLS**: match presented client cert thumbprint to token `cnf.x5t#S256`.
|
||||
3. Check **scopes**; optionally map to internal roles.
|
||||
4. Check **tenant** (`tid`) and **installation** (`inst`) as appropriate.
|
||||
5. For **Signer** only: require **both** OpTok and **PoE** in the request (enforced by Signer, not Authority).
|
||||
|
||||
---
|
||||
|
||||
## 12) Error surfaces & UX
|
||||
|
||||
* Token endpoint errors follow OAuth2 (`invalid_client`, `invalid_grant`, `invalid_scope`, `unauthorized_client`).
|
||||
* Resource servers use RFC 6750 style (`WWW-Authenticate: DPoP error="invalid_token", error_description="…", dpop_nonce="…" `).
|
||||
* For DPoP nonce challenges, clients retry with the server‑supplied nonce once.
|
||||
|
||||
---
|
||||
|
||||
## 13) Observability & audit
|
||||
|
||||
* **Metrics**:
|
||||
|
||||
* `authority.tokens_issued_total{grant,aud}`
|
||||
* `authority.dpop_validations_total{result}`
|
||||
* `authority.mtls_bindings_total{result}`
|
||||
* `authority.jwks_rotations_total`
|
||||
* `authority.errors_total{type}`
|
||||
* **Audit log** (immutable sink): token issuance (`sub`, `aud`, `scopes`, `tid`, `inst`, `cnf thumbprint`, `jti`), revocations, admin changes.
|
||||
* **Tracing**: token flows, DB reads, JWKS cache.
|
||||
|
||||
---
|
||||
|
||||
## 14) Configuration (YAML)
|
||||
|
||||
```yaml
|
||||
authority:
|
||||
issuer: "https://authority.internal"
|
||||
signing:
|
||||
enabled: true
|
||||
activeKeyId: "authority-signing-2025"
|
||||
keyPath: "../certificates/authority-signing-2025.pem"
|
||||
algorithm: "ES256"
|
||||
keySource: "file"
|
||||
security:
|
||||
rateLimiting:
|
||||
token:
|
||||
enabled: true
|
||||
permitLimit: 30
|
||||
window: "00:01:00"
|
||||
queueLimit: 0
|
||||
authorize:
|
||||
enabled: true
|
||||
permitLimit: 60
|
||||
window: "00:01:00"
|
||||
queueLimit: 10
|
||||
internal:
|
||||
enabled: false
|
||||
permitLimit: 5
|
||||
window: "00:01:00"
|
||||
queueLimit: 0
|
||||
senderConstraints:
|
||||
dpop:
|
||||
enabled: true
|
||||
allowedAlgorithms: [ "ES256", "ES384" ]
|
||||
proofLifetime: "00:02:00"
|
||||
allowedClockSkew: "00:00:30"
|
||||
replayWindow: "00:05:00"
|
||||
nonce:
|
||||
enabled: true
|
||||
ttl: "00:10:00"
|
||||
maxIssuancePerMinute: 120
|
||||
store: "redis"
|
||||
redisConnectionString: "redis://authority-redis:6379?ssl=false"
|
||||
requiredAudiences:
|
||||
- "signer"
|
||||
- "attestor"
|
||||
mtls:
|
||||
enabled: true
|
||||
requireChainValidation: true
|
||||
rotationGrace: "00:15:00"
|
||||
enforceForAudiences:
|
||||
- "signer"
|
||||
allowedSanTypes:
|
||||
- "dns"
|
||||
- "uri"
|
||||
allowedCertificateAuthorities:
|
||||
- "/etc/ssl/mtls/clients-ca.pem"
|
||||
clients:
|
||||
- clientId: scanner-web
|
||||
grantTypes: [ "client_credentials" ]
|
||||
audiences: [ "scanner" ]
|
||||
auth: { type: "private_key_jwt", jwkFile: "/secrets/scanner-web.jwk" }
|
||||
senderConstraint: "dpop"
|
||||
scopes: [ "scanner.scan", "scanner.export", "scanner.read" ]
|
||||
- clientId: signer
|
||||
grantTypes: [ "client_credentials" ]
|
||||
audiences: [ "signer" ]
|
||||
auth: { type: "mtls" }
|
||||
senderConstraint: "mtls"
|
||||
scopes: [ "signer.sign" ]
|
||||
- clientId: notify-web-dev
|
||||
grantTypes: [ "client_credentials" ]
|
||||
audiences: [ "notify.dev" ]
|
||||
auth: { type: "client_secret", secretFile: "/secrets/notify-web-dev.secret" }
|
||||
senderConstraint: "dpop"
|
||||
scopes: [ "notify.read", "notify.admin" ]
|
||||
- clientId: notify-web
|
||||
grantTypes: [ "client_credentials" ]
|
||||
audiences: [ "notify" ]
|
||||
auth: { type: "client_secret", secretFile: "/secrets/notify-web.secret" }
|
||||
senderConstraint: "dpop"
|
||||
scopes: [ "notify.read", "notify.admin" ]
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 15) Testing matrix
|
||||
|
||||
* **JWT validation**: wrong `aud`, expired `exp`, skewed `nbf`, stale `kid`.
|
||||
* **DPoP**: invalid `htu`/`htm`, replayed `jti`, stale `iat`, wrong `jkt`, nonce dance.
|
||||
* **mTLS**: wrong client cert, wrong CA, thumbprint mismatch.
|
||||
* **RBAC**: scope enforcement per audience; over‑privileged client denied.
|
||||
* **Rotation**: JWKS rotation while load‑testing; zero‑downtime verification.
|
||||
* **HA**: kill one Authority instance; verify issuance continues; JWKS served by peers.
|
||||
* **Performance**: 1k token issuance/sec on 2 cores with Redis enabled for jti caching.
|
||||
|
||||
---
|
||||
|
||||
## 16) Threat model & mitigations (summary)
|
||||
|
||||
| Threat | Vector | Mitigation |
|
||||
| ------------------- | ---------------- | ------------------------------------------------------------------------------------------ |
|
||||
| Token theft | Copy of JWT | **Short TTL**, **sender‑constraint** (DPoP/mTLS); replay blocked by `jti` cache and nonces |
|
||||
| Replay across hosts | Reuse DPoP proof | Enforce `htu`/`htm`, `iat` freshness, `jti` uniqueness; services may require **nonce** |
|
||||
| Impersonation | Fake client | mTLS or `private_key_jwt` with pinned JWK; client registration & rotation |
|
||||
| Key compromise | Signing key leak | HSM/KMS storage, key rotation, audit; emergency key revoke path; narrow token TTL |
|
||||
| Cross‑tenant abuse | Scope elevation | Enforce `aud`, `tid`, `inst` at issuance and resource servers |
|
||||
| Downgrade to bearer | Strip DPoP | Resource servers require DPoP/mTLS based on `aud`; reject bearer without `cnf` |
|
||||
|
||||
---
|
||||
|
||||
## 17) Deployment & HA
|
||||
|
||||
* **Stateless** microservice, containerized; run ≥ 2 replicas behind LB.
|
||||
* **DB**: HA Postgres (or MySQL) for clients/roles; **Redis** for device codes, DPoP nonces/jtis.
|
||||
* **Secrets**: mount client JWKs via K8s Secrets/HashiCorp Vault; signing keys via KMS.
|
||||
* **Backups**: DB daily; Redis not critical (ephemeral).
|
||||
* **Disaster recovery**: export/import of client registry; JWKS rehydrate from KMS.
|
||||
* **Compliance**: TLS audit; penetration testing for OIDC flows.
|
||||
|
||||
---
|
||||
|
||||
## 18) Implementation notes
|
||||
|
||||
* Reference stack: **.NET 10** + **OpenIddict 6** (or IdentityServer if licensed) with custom DPoP validator and mTLS binding middleware.
|
||||
* Keep the DPoP/JTI cache pluggable; allow Redis/Memcached.
|
||||
* Provide **client SDKs** for C# and Go: DPoP key mgmt, proof generation, nonce handling, token refresh helper.
|
||||
|
||||
---
|
||||
|
||||
## 19) Quick reference — wire examples
|
||||
|
||||
**Access token (payload excerpt)**
|
||||
|
||||
```json
|
||||
{
|
||||
"iss": "https://authority.internal",
|
||||
"sub": "scanner-web",
|
||||
"aud": "signer",
|
||||
"exp": 1760668800,
|
||||
"iat": 1760668620,
|
||||
"nbf": 1760668620,
|
||||
"jti": "9d9c3f01-6e1a-49f1-8f77-9b7e6f7e3c50",
|
||||
"scope": "signer.sign",
|
||||
"tid": "tenant-01",
|
||||
"inst": "install-7A2B",
|
||||
"cnf": { "jkt": "KcVb2V...base64url..." }
|
||||
}
|
||||
```
|
||||
|
||||
**DPoP proof header fields (for POST /sign/dsse)**
|
||||
|
||||
```json
|
||||
{
|
||||
"htu": "https://signer.internal/sign/dsse",
|
||||
"htm": "POST",
|
||||
"iat": 1760668620,
|
||||
"jti": "4b1c9b3c-8a95-4c58-8a92-9c6cfb4a6a0b"
|
||||
}
|
||||
```
|
||||
|
||||
Signer validates that `hash(JWK)` in the proof matches `cnf.jkt` in the token.
|
||||
|
||||
---
|
||||
|
||||
## 20) Rollout plan
|
||||
|
||||
1. **MVP**: Client Credentials (private_key_jwt + DPoP), JWKS, short OpToks, per‑audience scopes.
|
||||
2. **Add**: mTLS‑bound tokens for Signer/Attestor; device code for CLI; optional introspection.
|
||||
3. **Hardening**: DPoP nonce support; full audit pipeline; HA tuning.
|
||||
4. **UX**: Tenant/installation admin UI; role→scope editors; client bootstrap wizards.
|
||||
|
||||
@@ -1,406 +1,406 @@
|
||||
# component_architecture_cli.md — **Stella Ops CLI** (2025Q4)
|
||||
|
||||
> **Scope.** Implementation‑ready architecture for **Stella Ops CLI**: command surface, process model, auth (Authority/DPoP), integration with Scanner/Excititor/Concelier/Signer/Attestor, Buildx plug‑in management, offline kit behavior, packaging, observability, security posture, and CI ergonomics.
|
||||
|
||||
---
|
||||
|
||||
## 0) Mission & boundaries
|
||||
|
||||
**Mission.** Provide a **fast, deterministic, CI‑friendly** command‑line interface to drive Stella Ops workflows:
|
||||
|
||||
* Build‑time SBOM generation via **Buildx generator** orchestration.
|
||||
* Post‑build **scan/compose/diff/export** against **Scanner.WebService**.
|
||||
* **Policy** operations and **VEX/Vuln** data pulls (operator tasks).
|
||||
* **Verification** (attestation, referrers, signatures) for audits.
|
||||
* Air‑gapped/offline **kit** administration.
|
||||
|
||||
**Boundaries.**
|
||||
|
||||
* CLI **never** signs; it only calls **Signer**/**Attestor** via backend APIs when needed (e.g., `report --attest`).
|
||||
* CLI **does not** store long‑lived credentials beyond OS keychain; tokens are **short** (Authority OpToks).
|
||||
* Heavy work (scanning, merging, policy) is executed **server‑side** (Scanner/Excititor/Concelier).
|
||||
|
||||
---
|
||||
|
||||
## 1) Solution layout & runtime form
|
||||
|
||||
```
|
||||
src/
|
||||
├─ StellaOps.Cli/ # net10.0 (Native AOT) single binary
|
||||
├─ StellaOps.Cli.Core/ # verb plumbing, config, HTTP, auth
|
||||
├─ StellaOps.Cli.Plugins/ # optional verbs packaged as plugins
|
||||
├─ StellaOps.Cli.Tests/ # unit + golden-output tests
|
||||
└─ packaging/
|
||||
├─ msix / msi / deb / rpm / brew formula
|
||||
└─ scoop manifest / winget manifest
|
||||
```
|
||||
|
||||
**Language/runtime**: .NET 10 **Native AOT** for speed/startup; Linux builds use **musl** static when possible.
|
||||
|
||||
**Plug-in verbs.** Non-core verbs (Excititor, runtime helpers, future integrations) ship as restart-time plug-ins under `plugins/cli/**` with manifest descriptors. The launcher loads plug-ins on startup; hot reloading is intentionally unsupported. The inaugural bundle, `StellaOps.Cli.Plugins.NonCore`, packages the Excititor, runtime, and offline-kit command groups and publishes its manifest at `plugins/cli/StellaOps.Cli.Plugins.NonCore/`.
|
||||
|
||||
**OS targets**: linux‑x64/arm64, windows‑x64/arm64, macOS‑x64/arm64.
|
||||
|
||||
---
|
||||
|
||||
## 2) Command surface (verbs)
|
||||
|
||||
> All verbs default to **JSON** output when `--json` is set (CI mode). Human output is concise, deterministic.
|
||||
|
||||
### 2.1 Auth & profile
|
||||
|
||||
* `auth login`
|
||||
|
||||
* Modes: **device‑code** (default), **client‑credentials** (service principal).
|
||||
* Produces **Authority** access token (OpTok) + stores **DPoP** keypair in OS keychain.
|
||||
* `auth status` — show current issuer, subject, audiences, expiry.
|
||||
* `auth logout` — wipe cached tokens/keys.
|
||||
|
||||
### 2.2 Build‑time SBOM (Buildx)
|
||||
|
||||
* `buildx install` — install/update the **StellaOps.Scanner.Sbomer.BuildXPlugin** on the host.
|
||||
* `buildx verify` — ensure generator is usable.
|
||||
* `buildx build` — thin wrapper around `docker buildx build --attest=type=sbom,generator=stellaops/sbom-indexer` with convenience flags:
|
||||
|
||||
* `--attest` (request Signer/Attestor via backend post‑push)
|
||||
* `--provenance` pass‑through (optional)
|
||||
|
||||
### 2.3 Scanning & artifacts
|
||||
|
||||
* `scan image <ref|digest>`
|
||||
|
||||
* Options: `--force`, `--wait`, `--view=inventory|usage|both`, `--format=cdx-json|cdx-pb|spdx-json`, `--attest` (ask backend to sign/log).
|
||||
* Streams progress; exits early unless `--wait`.
|
||||
* `diff image --old <digest> --new <digest> [--view ...]` — show layer‑attributed changes.
|
||||
* `export sbom <digest> [--view ... --format ... --out file]` — download artifact.
|
||||
* `report final <digest> [--policy-revision ... --attest]` — request PASS/FAIL report from backend (policy+vex) and optional attestation.
|
||||
|
||||
### 2.4 Policy & data
|
||||
|
||||
* `policy get/set/apply` — fetch active policy, apply staged policy, compute digest.
|
||||
* `concelier export` — trigger/export canonical JSON or Trivy DB (admin).
|
||||
* `excititor export` — trigger/export consensus/raw claims (admin).
|
||||
|
||||
### 2.5 Verification
|
||||
|
||||
* `verify attestation --uuid <rekor-uuid> | --artifact <sha256> | --bundle <path>` — call **Attestor /verify** and print proof summary.
|
||||
* `verify referrers <digest>` — ask **Signer /verify/referrers** (is image Stella‑signed?).
|
||||
* `verify image-signature <ref|digest>` — standalone cosign verification (optional, local).
|
||||
|
||||
### 2.6 Runtime (Zastava helper)
|
||||
|
||||
* `runtime policy test --image/-i <digest> [--file <path> --ns <name> --label key=value --json]` — ask backend `/policy/runtime` like the webhook would (accepts multiple `--image`, comma/space lists, or stdin pipelines).
|
||||
|
||||
### 2.7 Offline kit
|
||||
|
||||
* `offline kit pull` — fetch latest **Concelier JSON + Trivy DB + Excititor exports** as a tarball from a mirror.
|
||||
* `offline kit import <tar>` — upload the kit to on‑prem services (Concelier/Excititor).
|
||||
* `offline kit status` — list current seed versions.
|
||||
|
||||
### 2.8 Utilities
|
||||
|
||||
* `config set/get` — endpoint & defaults.
|
||||
* `whoami` — short auth display.
|
||||
* `version` — CLI + protocol versions; release channel.
|
||||
|
||||
### 2.9 Aggregation-only guard helpers
|
||||
|
||||
* `sources ingest --dry-run --source <id> --input <path|uri> [--tenant ... --format table|json --output file]`
|
||||
|
||||
* Normalises documents (handles gzip/base64), posts them to the backend `aoc/ingest/dry-run` route, and exits non-zero when guard violations are detected.
|
||||
* Defaults to table output with ANSI colour; `--json`/`--output` produce deterministic JSON for CI pipelines.
|
||||
|
||||
* `aoc verify [--since <ISO8601|duration>] [--limit <count>] [--sources list] [--codes list] [--format table|json] [--export file] [--tenant id] [--no-color]`
|
||||
|
||||
* Replays guard checks against stored raw documents. Maps backend `ERR_AOC_00x` codes onto deterministic exit codes so CI can block regressions.
|
||||
* Supports pagination hints (`--limit`, `--since`), tenant scoping via `--tenant` or `STELLA_TENANT`, and JSON exports for evidence lockers.
|
||||
|
||||
---
|
||||
|
||||
## 3) AuthN: Authority + DPoP
|
||||
|
||||
### 3.1 Token acquisition
|
||||
|
||||
* **Device‑code**: the CLI opens an OIDC device code flow against **Authority**; the browser login is optional for service principals.
|
||||
* **Client‑credentials**: service principals use **private_key_jwt** or **mTLS** to get tokens.
|
||||
|
||||
### 3.2 DPoP key management
|
||||
|
||||
* On first login, the CLI generates an **ephemeral JWK** (Ed25519) and stores it in the **OS keychain** (Keychain/DPAPI/KWallet/Gnome Keyring).
|
||||
* Every request to backend services includes a **DPoP proof**; CLI refreshes tokens as needed.
|
||||
|
||||
### 3.3 Multi‑audience & scopes
|
||||
|
||||
* CLI requests **audiences** as needed per verb:
|
||||
|
||||
* `scanner` for scan/export/report/diff
|
||||
* `signer` (indirect; usually backend calls Signer)
|
||||
* `attestor` for verify
|
||||
* `concelier`/`excititor` for admin verbs
|
||||
|
||||
CLI rejects verbs if required scopes are missing.
|
||||
|
||||
---
|
||||
|
||||
## 4) Process model & reliability
|
||||
|
||||
### 4.1 HTTP client
|
||||
|
||||
* Single **http2** client with connection pooling, DNS pinning, retry/backoff (idempotent GET/POST marked safe).
|
||||
* **DPoP nonce** handling: on `401` with nonce challenge, CLI replays once.
|
||||
|
||||
### 4.2 Streaming
|
||||
|
||||
* `scan` and `report` support **server‑sent JSON lines** (progress events).
|
||||
* `--json` prints machine events; human mode shows compact spinners and crucial updates only.
|
||||
|
||||
### 4.3 Exit codes (CI‑safe)
|
||||
|
||||
| Code | Meaning |
|
||||
| ---- | ------------------------------------------- |
|
||||
| 0 | Success |
|
||||
| 2 | Policy fail (final report verdict=fail) |
|
||||
| 3 | Verification failed (attestation/signature) |
|
||||
| 4 | Auth error (invalid/missing token/DPoP) |
|
||||
| 5 | Resource not found (image/SBOM) |
|
||||
| 6 | Rate limited / quota exceeded |
|
||||
| 7 | Backend unavailable (retryable) |
|
||||
| 9 | Invalid arguments |
|
||||
| 11–17 | Aggregation-only guard violation (`ERR_AOC_00x`) |
|
||||
| 18 | Verification truncated (increase `--limit`) |
|
||||
| 70 | Transport/authentication failure |
|
||||
| 71 | CLI usage error (missing tenant, invalid cursor) |
|
||||
|
||||
---
|
||||
|
||||
## 5) Configuration model
|
||||
|
||||
**Precedence:** CLI flags → env vars → config file → defaults.
|
||||
|
||||
**Config file**: `${XDG_CONFIG_HOME}/stellaops/config.yaml` (Windows: `%APPDATA%\StellaOps\config.yaml`)
|
||||
|
||||
```yaml
|
||||
cli:
|
||||
authority: "https://authority.internal"
|
||||
backend:
|
||||
scanner: "https://scanner-web.internal"
|
||||
attestor: "https://attestor.internal"
|
||||
concelier: "https://concelier-web.internal"
|
||||
excititor: "https://excititor-web.internal"
|
||||
auth:
|
||||
audienceDefault: "scanner"
|
||||
deviceCode: true
|
||||
output:
|
||||
json: false
|
||||
color: auto
|
||||
tls:
|
||||
caBundle: "/etc/ssl/certs/ca-bundle.crt"
|
||||
offline:
|
||||
kitMirror: "s3://mirror/stellaops-kit"
|
||||
```
|
||||
|
||||
Environment variables: `STELLAOPS_AUTHORITY`, `STELLAOPS_SCANNER_URL`, etc.
|
||||
|
||||
---
|
||||
|
||||
## 6) Buildx generator orchestration
|
||||
|
||||
* `buildx install` locates the Docker root directory, writes the **generator** plugin manifest, and pulls `stellaops/sbom-indexer` image (pinned digest).
|
||||
* `buildx build` wrapper injects:
|
||||
|
||||
* `--attest=type=sbom,generator=stellaops/sbom-indexer`
|
||||
* `--label org.stellaops.request=sbom`
|
||||
* Post‑build: CLI optionally calls **Scanner.WebService** to **verify referrers**, **compose** image SBOMs, and **attest** via Signer/Attestor.
|
||||
|
||||
**Detection**: If Buildx or generator unavailable, CLI falls back to **post‑build scan** with a warning.
|
||||
|
||||
---
|
||||
|
||||
## 7) Artifact handling
|
||||
|
||||
* **Downloads** (`export sbom`, `report final`): stream to file; compute sha256 on the fly; write sidecar `.sha256` and optional **verification bundle** (if `--bundle`).
|
||||
* **Uploads** (`offline kit import`): chunked upload; retry on transient errors; show progress bar (unless `--json`).
|
||||
|
||||
---
|
||||
|
||||
## 8) Security posture
|
||||
|
||||
* **DPoP private keys** stored in **OS keychain**; metadata cached in config.
|
||||
* **No plaintext tokens** on disk; short‑lived **OpToks** held in memory.
|
||||
* **TLS**: verify backend certificates; allow custom CA bundle for on‑prem.
|
||||
* **Redaction**: CLI logs remove `Authorization`, DPoP headers, PoE tokens.
|
||||
* **Supply chain**: CLI distribution binaries are **cosign‑signed**; `stellaops version --verify` checks its own signature.
|
||||
|
||||
---
|
||||
|
||||
## 9) Observability
|
||||
|
||||
* `--verbose` adds request IDs, timings, and retry traces.
|
||||
* **Metrics** (optional, disabled by default): Prometheus text file exporter for local monitoring in long‑running agents.
|
||||
* **Structured logs** (`--json`): per‑event JSON lines with `ts`, `verb`, `status`, `latencyMs`.
|
||||
|
||||
---
|
||||
|
||||
## 10) Performance targets
|
||||
|
||||
* Startup ≤ **20 ms** (AOT).
|
||||
* `scan image` request/response overhead ≤ **5 ms** (excluding server work).
|
||||
* Buildx wrapper overhead negligible (<1 ms).
|
||||
* Large artifact download (100 MB) sustained ≥ **80 MB/s** on local networks.
|
||||
|
||||
---
|
||||
|
||||
## 11) Tests & golden outputs
|
||||
|
||||
* **Unit tests**: argument parsing, config precedence, URL resolution, DPoP proof creation.
|
||||
* **Integration tests** (Testcontainers): mock Authority/Scanner/Attestor; CI pipeline with fake registry.
|
||||
* **Golden outputs**: verb snapshots for `--json` across OSes; kept in `tests/golden/…`.
|
||||
* **Contract tests**: ensure API shapes match service OpenAPI; fail build if incompatible.
|
||||
|
||||
---
|
||||
|
||||
## 12) Error envelopes (human + JSON)
|
||||
|
||||
**Human:**
|
||||
|
||||
```
|
||||
✖ Policy FAIL: 3 high, 1 critical (VEX suppressed 12)
|
||||
- pkg:rpm/openssl (CVE-2025-12345) — affected (vendor) — fixed in 3.0.14
|
||||
- pkg:npm/lodash (GHSA-xxxx) — affected — no fix
|
||||
See: https://ui.internal/scans/sha256:...
|
||||
Exit code: 2
|
||||
```
|
||||
|
||||
**JSON (`--json`):**
|
||||
|
||||
```json
|
||||
{ "event":"report", "status":"fail", "critical":1, "high":3, "url":"https://ui..." }
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 13) Admin & advanced flags
|
||||
|
||||
* `--authority`, `--scanner`, `--attestor`, `--concelier`, `--excititor` override config URLs.
|
||||
* `--no-color`, `--quiet`, `--json`.
|
||||
* `--timeout`, `--retries`, `--retry-backoff-ms`.
|
||||
* `--ca-bundle`, `--insecure` (dev only; prints warning).
|
||||
* `--trace` (dump HTTP traces to file; scrubbed).
|
||||
|
||||
---
|
||||
|
||||
## 14) Interop with other tools
|
||||
|
||||
* Emits **CycloneDX Protobuf** directly to stdout when `export sbom --format cdx-pb --out -`.
|
||||
* Pipes to `jq`/`yq` cleanly in JSON mode.
|
||||
* Can act as a **credential helper** for scripts: `stellaops auth token --aud scanner` prints a one‑shot token for curl.
|
||||
|
||||
---
|
||||
|
||||
## 15) Packaging & distribution
|
||||
|
||||
* **Installers**: deb/rpm (postinst registers completions), Homebrew, Scoop, Winget, MSI/MSIX.
|
||||
* **Shell completions**: bash/zsh/fish/pwsh.
|
||||
* **Update channel**: `stellaops self-update` (optional) fetches cosign‑signed release manifest; corporate environments can disable.
|
||||
|
||||
---
|
||||
|
||||
## 16) Security hard lines
|
||||
|
||||
* Refuse to print token values; redact Authorization headers in verbose output.
|
||||
* Disallow `--insecure` unless `STELLAOPS_CLI_ALLOW_INSECURE=1` set (double opt‑in).
|
||||
* Enforce **short token TTL**; refresh proactively when <30 s left.
|
||||
* Device‑code cache binding to **machine** and **user** (protect against copy to other machines).
|
||||
|
||||
---
|
||||
|
||||
## 17) Wire sequences
|
||||
|
||||
**A) Scan & wait with attestation**
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
autonumber
|
||||
participant CLI
|
||||
participant Auth as Authority
|
||||
participant SW as Scanner.WebService
|
||||
participant SG as Signer
|
||||
participant AT as Attestor
|
||||
|
||||
CLI->>Auth: device code flow (DPoP)
|
||||
Auth-->>CLI: OpTok (aud=scanner)
|
||||
|
||||
CLI->>SW: POST /scans { imageRef, attest:true }
|
||||
SW-->>CLI: { scanId }
|
||||
CLI->>SW: GET /scans/{id} (poll)
|
||||
SW-->>CLI: { status: completed, artifacts, rekor? } # if attested
|
||||
|
||||
alt attestation pending
|
||||
SW->>SG: POST /sign/dsse (server-side)
|
||||
SG-->>SW: DSSE
|
||||
SW->>AT: POST /rekor/entries
|
||||
AT-->>SW: { uuid, proof }
|
||||
end
|
||||
|
||||
CLI->>SW: GET /sboms/<digest>?format=cdx-pb&view=usage
|
||||
SW-->>CLI: bytes
|
||||
```
|
||||
|
||||
**B) Verify attestation by artifact**
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
autonumber
|
||||
participant CLI
|
||||
participant AT as Attestor
|
||||
|
||||
CLI->>AT: POST /rekor/verify { artifactSha256 }
|
||||
AT-->>CLI: { ok:true, uuid, index, logURL }
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 18) Roadmap (CLI)
|
||||
|
||||
* `scan fs <path>` (local filesystem tree) → upload to backend for analysis.
|
||||
* `policy test --sbom <file>` (simulate policy results offline using local policy bundle).
|
||||
* `runtime capture` (developer mode) — capture small `/proc/<pid>/maps` for troubleshooting.
|
||||
* Pluggable output renderers for SARIF/HTML (admin‑controlled).
|
||||
|
||||
---
|
||||
|
||||
## 19) Example CI snippets
|
||||
|
||||
**GitHub Actions (post‑build)**
|
||||
|
||||
```yaml
|
||||
- name: Login (device code w/ OIDC broker)
|
||||
run: stellaops auth login --json --authority ${{ secrets.AUTHORITY_URL }}
|
||||
|
||||
- name: Scan
|
||||
run: stellaops scan image ${{ steps.build.outputs.digest }} --wait --json
|
||||
|
||||
- name: Export (usage view, protobuf)
|
||||
run: stellaops export sbom ${{ steps.build.outputs.digest }} --view usage --format cdx-pb --out sbom.pb
|
||||
|
||||
- name: Verify attestation
|
||||
run: stellaops verify attestation --artifact $(sha256sum sbom.pb | cut -d' ' -f1) --json
|
||||
```
|
||||
|
||||
**GitLab (buildx generator)**
|
||||
|
||||
```yaml
|
||||
script:
|
||||
- stellaops buildx install
|
||||
- docker buildx build --attest=type=sbom,generator=stellaops/sbom-indexer -t $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA .
|
||||
- stellaops scan image $CI_REGISTRY_IMAGE@$IMAGE_DIGEST --wait --json
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 20) Test matrix (OS/arch)
|
||||
|
||||
* Linux: ubuntu‑20.04/22.04/24.04 (x64, arm64), alpine (musl).
|
||||
* macOS: 13–15 (x64, arm64).
|
||||
* Windows: 10/11, Server 2019/2022 (x64, arm64).
|
||||
* Docker engines: Docker Desktop, containerd‑based runners.
|
||||
# component_architecture_cli.md — **Stella Ops CLI** (2025Q4)
|
||||
|
||||
> **Scope.** Implementation‑ready architecture for **Stella Ops CLI**: command surface, process model, auth (Authority/DPoP), integration with Scanner/Excititor/Concelier/Signer/Attestor, Buildx plug‑in management, offline kit behavior, packaging, observability, security posture, and CI ergonomics.
|
||||
|
||||
---
|
||||
|
||||
## 0) Mission & boundaries
|
||||
|
||||
**Mission.** Provide a **fast, deterministic, CI‑friendly** command‑line interface to drive Stella Ops workflows:
|
||||
|
||||
* Build‑time SBOM generation via **Buildx generator** orchestration.
|
||||
* Post‑build **scan/compose/diff/export** against **Scanner.WebService**.
|
||||
* **Policy** operations and **VEX/Vuln** data pulls (operator tasks).
|
||||
* **Verification** (attestation, referrers, signatures) for audits.
|
||||
* Air‑gapped/offline **kit** administration.
|
||||
|
||||
**Boundaries.**
|
||||
|
||||
* CLI **never** signs; it only calls **Signer**/**Attestor** via backend APIs when needed (e.g., `report --attest`).
|
||||
* CLI **does not** store long‑lived credentials beyond OS keychain; tokens are **short** (Authority OpToks).
|
||||
* Heavy work (scanning, merging, policy) is executed **server‑side** (Scanner/Excititor/Concelier).
|
||||
|
||||
---
|
||||
|
||||
## 1) Solution layout & runtime form
|
||||
|
||||
```
|
||||
src/
|
||||
├─ StellaOps.Cli/ # net10.0 (Native AOT) single binary
|
||||
├─ StellaOps.Cli.Core/ # verb plumbing, config, HTTP, auth
|
||||
├─ StellaOps.Cli.Plugins/ # optional verbs packaged as plugins
|
||||
├─ StellaOps.Cli.Tests/ # unit + golden-output tests
|
||||
└─ packaging/
|
||||
├─ msix / msi / deb / rpm / brew formula
|
||||
└─ scoop manifest / winget manifest
|
||||
```
|
||||
|
||||
**Language/runtime**: .NET 10 **Native AOT** for speed/startup; Linux builds use **musl** static when possible.
|
||||
|
||||
**Plug-in verbs.** Non-core verbs (Excititor, runtime helpers, future integrations) ship as restart-time plug-ins under `plugins/cli/**` with manifest descriptors. The launcher loads plug-ins on startup; hot reloading is intentionally unsupported. The inaugural bundle, `StellaOps.Cli.Plugins.NonCore`, packages the Excititor, runtime, and offline-kit command groups and publishes its manifest at `plugins/cli/StellaOps.Cli.Plugins.NonCore/`.
|
||||
|
||||
**OS targets**: linux‑x64/arm64, windows‑x64/arm64, macOS‑x64/arm64.
|
||||
|
||||
---
|
||||
|
||||
## 2) Command surface (verbs)
|
||||
|
||||
> All verbs default to **JSON** output when `--json` is set (CI mode). Human output is concise, deterministic.
|
||||
|
||||
### 2.1 Auth & profile
|
||||
|
||||
* `auth login`
|
||||
|
||||
* Modes: **device‑code** (default), **client‑credentials** (service principal).
|
||||
* Produces **Authority** access token (OpTok) + stores **DPoP** keypair in OS keychain.
|
||||
* `auth status` — show current issuer, subject, audiences, expiry.
|
||||
* `auth logout` — wipe cached tokens/keys.
|
||||
|
||||
### 2.2 Build‑time SBOM (Buildx)
|
||||
|
||||
* `buildx install` — install/update the **StellaOps.Scanner.Sbomer.BuildXPlugin** on the host.
|
||||
* `buildx verify` — ensure generator is usable.
|
||||
* `buildx build` — thin wrapper around `docker buildx build --attest=type=sbom,generator=stellaops/sbom-indexer` with convenience flags:
|
||||
|
||||
* `--attest` (request Signer/Attestor via backend post‑push)
|
||||
* `--provenance` pass‑through (optional)
|
||||
|
||||
### 2.3 Scanning & artifacts
|
||||
|
||||
* `scan image <ref|digest>`
|
||||
|
||||
* Options: `--force`, `--wait`, `--view=inventory|usage|both`, `--format=cdx-json|cdx-pb|spdx-json`, `--attest` (ask backend to sign/log).
|
||||
* Streams progress; exits early unless `--wait`.
|
||||
* `diff image --old <digest> --new <digest> [--view ...]` — show layer‑attributed changes.
|
||||
* `export sbom <digest> [--view ... --format ... --out file]` — download artifact.
|
||||
* `report final <digest> [--policy-revision ... --attest]` — request PASS/FAIL report from backend (policy+vex) and optional attestation.
|
||||
|
||||
### 2.4 Policy & data
|
||||
|
||||
* `policy get/set/apply` — fetch active policy, apply staged policy, compute digest.
|
||||
* `concelier export` — trigger/export canonical JSON or Trivy DB (admin).
|
||||
* `excititor export` — trigger/export consensus/raw claims (admin).
|
||||
|
||||
### 2.5 Verification
|
||||
|
||||
* `verify attestation --uuid <rekor-uuid> | --artifact <sha256> | --bundle <path>` — call **Attestor /verify** and print proof summary.
|
||||
* `verify referrers <digest>` — ask **Signer /verify/referrers** (is image Stella‑signed?).
|
||||
* `verify image-signature <ref|digest>` — standalone cosign verification (optional, local).
|
||||
|
||||
### 2.6 Runtime (Zastava helper)
|
||||
|
||||
* `runtime policy test --image/-i <digest> [--file <path> --ns <name> --label key=value --json]` — ask backend `/policy/runtime` like the webhook would (accepts multiple `--image`, comma/space lists, or stdin pipelines).
|
||||
|
||||
### 2.7 Offline kit
|
||||
|
||||
* `offline kit pull` — fetch latest **Concelier JSON + Trivy DB + Excititor exports** as a tarball from a mirror.
|
||||
* `offline kit import <tar>` — upload the kit to on‑prem services (Concelier/Excititor).
|
||||
* `offline kit status` — list current seed versions.
|
||||
|
||||
### 2.8 Utilities
|
||||
|
||||
* `config set/get` — endpoint & defaults.
|
||||
* `whoami` — short auth display.
|
||||
* `version` — CLI + protocol versions; release channel.
|
||||
|
||||
### 2.9 Aggregation-only guard helpers
|
||||
|
||||
* `sources ingest --dry-run --source <id> --input <path|uri> [--tenant ... --format table|json --output file]`
|
||||
|
||||
* Normalises documents (handles gzip/base64), posts them to the backend `aoc/ingest/dry-run` route, and exits non-zero when guard violations are detected.
|
||||
* Defaults to table output with ANSI colour; `--json`/`--output` produce deterministic JSON for CI pipelines.
|
||||
|
||||
* `aoc verify [--since <ISO8601|duration>] [--limit <count>] [--sources list] [--codes list] [--format table|json] [--export file] [--tenant id] [--no-color]`
|
||||
|
||||
* Replays guard checks against stored raw documents. Maps backend `ERR_AOC_00x` codes onto deterministic exit codes so CI can block regressions.
|
||||
* Supports pagination hints (`--limit`, `--since`), tenant scoping via `--tenant` or `STELLA_TENANT`, and JSON exports for evidence lockers.
|
||||
|
||||
---
|
||||
|
||||
## 3) AuthN: Authority + DPoP
|
||||
|
||||
### 3.1 Token acquisition
|
||||
|
||||
* **Device‑code**: the CLI opens an OIDC device code flow against **Authority**; the browser login is optional for service principals.
|
||||
* **Client‑credentials**: service principals use **private_key_jwt** or **mTLS** to get tokens.
|
||||
|
||||
### 3.2 DPoP key management
|
||||
|
||||
* On first login, the CLI generates an **ephemeral JWK** (Ed25519) and stores it in the **OS keychain** (Keychain/DPAPI/KWallet/Gnome Keyring).
|
||||
* Every request to backend services includes a **DPoP proof**; CLI refreshes tokens as needed.
|
||||
|
||||
### 3.3 Multi‑audience & scopes
|
||||
|
||||
* CLI requests **audiences** as needed per verb:
|
||||
|
||||
* `scanner` for scan/export/report/diff
|
||||
* `signer` (indirect; usually backend calls Signer)
|
||||
* `attestor` for verify
|
||||
* `concelier`/`excititor` for admin verbs
|
||||
|
||||
CLI rejects verbs if required scopes are missing.
|
||||
|
||||
---
|
||||
|
||||
## 4) Process model & reliability
|
||||
|
||||
### 4.1 HTTP client
|
||||
|
||||
* Single **http2** client with connection pooling, DNS pinning, retry/backoff (idempotent GET/POST marked safe).
|
||||
* **DPoP nonce** handling: on `401` with nonce challenge, CLI replays once.
|
||||
|
||||
### 4.2 Streaming
|
||||
|
||||
* `scan` and `report` support **server‑sent JSON lines** (progress events).
|
||||
* `--json` prints machine events; human mode shows compact spinners and crucial updates only.
|
||||
|
||||
### 4.3 Exit codes (CI‑safe)
|
||||
|
||||
| Code | Meaning |
|
||||
| ---- | ------------------------------------------- |
|
||||
| 0 | Success |
|
||||
| 2 | Policy fail (final report verdict=fail) |
|
||||
| 3 | Verification failed (attestation/signature) |
|
||||
| 4 | Auth error (invalid/missing token/DPoP) |
|
||||
| 5 | Resource not found (image/SBOM) |
|
||||
| 6 | Rate limited / quota exceeded |
|
||||
| 7 | Backend unavailable (retryable) |
|
||||
| 9 | Invalid arguments |
|
||||
| 11–17 | Aggregation-only guard violation (`ERR_AOC_00x`) |
|
||||
| 18 | Verification truncated (increase `--limit`) |
|
||||
| 70 | Transport/authentication failure |
|
||||
| 71 | CLI usage error (missing tenant, invalid cursor) |
|
||||
|
||||
---
|
||||
|
||||
## 5) Configuration model
|
||||
|
||||
**Precedence:** CLI flags → env vars → config file → defaults.
|
||||
|
||||
**Config file**: `${XDG_CONFIG_HOME}/stellaops/config.yaml` (Windows: `%APPDATA%\StellaOps\config.yaml`)
|
||||
|
||||
```yaml
|
||||
cli:
|
||||
authority: "https://authority.internal"
|
||||
backend:
|
||||
scanner: "https://scanner-web.internal"
|
||||
attestor: "https://attestor.internal"
|
||||
concelier: "https://concelier-web.internal"
|
||||
excititor: "https://excititor-web.internal"
|
||||
auth:
|
||||
audienceDefault: "scanner"
|
||||
deviceCode: true
|
||||
output:
|
||||
json: false
|
||||
color: auto
|
||||
tls:
|
||||
caBundle: "/etc/ssl/certs/ca-bundle.crt"
|
||||
offline:
|
||||
kitMirror: "s3://mirror/stellaops-kit"
|
||||
```
|
||||
|
||||
Environment variables: `STELLAOPS_AUTHORITY`, `STELLAOPS_SCANNER_URL`, etc.
|
||||
|
||||
---
|
||||
|
||||
## 6) Buildx generator orchestration
|
||||
|
||||
* `buildx install` locates the Docker root directory, writes the **generator** plugin manifest, and pulls `stellaops/sbom-indexer` image (pinned digest).
|
||||
* `buildx build` wrapper injects:
|
||||
|
||||
* `--attest=type=sbom,generator=stellaops/sbom-indexer`
|
||||
* `--label org.stellaops.request=sbom`
|
||||
* Post‑build: CLI optionally calls **Scanner.WebService** to **verify referrers**, **compose** image SBOMs, and **attest** via Signer/Attestor.
|
||||
|
||||
**Detection**: If Buildx or generator unavailable, CLI falls back to **post‑build scan** with a warning.
|
||||
|
||||
---
|
||||
|
||||
## 7) Artifact handling
|
||||
|
||||
* **Downloads** (`export sbom`, `report final`): stream to file; compute sha256 on the fly; write sidecar `.sha256` and optional **verification bundle** (if `--bundle`).
|
||||
* **Uploads** (`offline kit import`): chunked upload; retry on transient errors; show progress bar (unless `--json`).
|
||||
|
||||
---
|
||||
|
||||
## 8) Security posture
|
||||
|
||||
* **DPoP private keys** stored in **OS keychain**; metadata cached in config.
|
||||
* **No plaintext tokens** on disk; short‑lived **OpToks** held in memory.
|
||||
* **TLS**: verify backend certificates; allow custom CA bundle for on‑prem.
|
||||
* **Redaction**: CLI logs remove `Authorization`, DPoP headers, PoE tokens.
|
||||
* **Supply chain**: CLI distribution binaries are **cosign‑signed**; `stellaops version --verify` checks its own signature.
|
||||
|
||||
---
|
||||
|
||||
## 9) Observability
|
||||
|
||||
* `--verbose` adds request IDs, timings, and retry traces.
|
||||
* **Metrics** (optional, disabled by default): Prometheus text file exporter for local monitoring in long‑running agents.
|
||||
* **Structured logs** (`--json`): per‑event JSON lines with `ts`, `verb`, `status`, `latencyMs`.
|
||||
|
||||
---
|
||||
|
||||
## 10) Performance targets
|
||||
|
||||
* Startup ≤ **20 ms** (AOT).
|
||||
* `scan image` request/response overhead ≤ **5 ms** (excluding server work).
|
||||
* Buildx wrapper overhead negligible (<1 ms).
|
||||
* Large artifact download (100 MB) sustained ≥ **80 MB/s** on local networks.
|
||||
|
||||
---
|
||||
|
||||
## 11) Tests & golden outputs
|
||||
|
||||
* **Unit tests**: argument parsing, config precedence, URL resolution, DPoP proof creation.
|
||||
* **Integration tests** (Testcontainers): mock Authority/Scanner/Attestor; CI pipeline with fake registry.
|
||||
* **Golden outputs**: verb snapshots for `--json` across OSes; kept in `tests/golden/…`.
|
||||
* **Contract tests**: ensure API shapes match service OpenAPI; fail build if incompatible.
|
||||
|
||||
---
|
||||
|
||||
## 12) Error envelopes (human + JSON)
|
||||
|
||||
**Human:**
|
||||
|
||||
```
|
||||
✖ Policy FAIL: 3 high, 1 critical (VEX suppressed 12)
|
||||
- pkg:rpm/openssl (CVE-2025-12345) — affected (vendor) — fixed in 3.0.14
|
||||
- pkg:npm/lodash (GHSA-xxxx) — affected — no fix
|
||||
See: https://ui.internal/scans/sha256:...
|
||||
Exit code: 2
|
||||
```
|
||||
|
||||
**JSON (`--json`):**
|
||||
|
||||
```json
|
||||
{ "event":"report", "status":"fail", "critical":1, "high":3, "url":"https://ui..." }
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 13) Admin & advanced flags
|
||||
|
||||
* `--authority`, `--scanner`, `--attestor`, `--concelier`, `--excititor` override config URLs.
|
||||
* `--no-color`, `--quiet`, `--json`.
|
||||
* `--timeout`, `--retries`, `--retry-backoff-ms`.
|
||||
* `--ca-bundle`, `--insecure` (dev only; prints warning).
|
||||
* `--trace` (dump HTTP traces to file; scrubbed).
|
||||
|
||||
---
|
||||
|
||||
## 14) Interop with other tools
|
||||
|
||||
* Emits **CycloneDX Protobuf** directly to stdout when `export sbom --format cdx-pb --out -`.
|
||||
* Pipes to `jq`/`yq` cleanly in JSON mode.
|
||||
* Can act as a **credential helper** for scripts: `stellaops auth token --aud scanner` prints a one‑shot token for curl.
|
||||
|
||||
---
|
||||
|
||||
## 15) Packaging & distribution
|
||||
|
||||
* **Installers**: deb/rpm (postinst registers completions), Homebrew, Scoop, Winget, MSI/MSIX.
|
||||
* **Shell completions**: bash/zsh/fish/pwsh.
|
||||
* **Update channel**: `stellaops self-update` (optional) fetches cosign‑signed release manifest; corporate environments can disable.
|
||||
|
||||
---
|
||||
|
||||
## 16) Security hard lines
|
||||
|
||||
* Refuse to print token values; redact Authorization headers in verbose output.
|
||||
* Disallow `--insecure` unless `STELLAOPS_CLI_ALLOW_INSECURE=1` set (double opt‑in).
|
||||
* Enforce **short token TTL**; refresh proactively when <30 s left.
|
||||
* Device‑code cache binding to **machine** and **user** (protect against copy to other machines).
|
||||
|
||||
---
|
||||
|
||||
## 17) Wire sequences
|
||||
|
||||
**A) Scan & wait with attestation**
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
autonumber
|
||||
participant CLI
|
||||
participant Auth as Authority
|
||||
participant SW as Scanner.WebService
|
||||
participant SG as Signer
|
||||
participant AT as Attestor
|
||||
|
||||
CLI->>Auth: device code flow (DPoP)
|
||||
Auth-->>CLI: OpTok (aud=scanner)
|
||||
|
||||
CLI->>SW: POST /scans { imageRef, attest:true }
|
||||
SW-->>CLI: { scanId }
|
||||
CLI->>SW: GET /scans/{id} (poll)
|
||||
SW-->>CLI: { status: completed, artifacts, rekor? } # if attested
|
||||
|
||||
alt attestation pending
|
||||
SW->>SG: POST /sign/dsse (server-side)
|
||||
SG-->>SW: DSSE
|
||||
SW->>AT: POST /rekor/entries
|
||||
AT-->>SW: { uuid, proof }
|
||||
end
|
||||
|
||||
CLI->>SW: GET /sboms/<digest>?format=cdx-pb&view=usage
|
||||
SW-->>CLI: bytes
|
||||
```
|
||||
|
||||
**B) Verify attestation by artifact**
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
autonumber
|
||||
participant CLI
|
||||
participant AT as Attestor
|
||||
|
||||
CLI->>AT: POST /rekor/verify { artifactSha256 }
|
||||
AT-->>CLI: { ok:true, uuid, index, logURL }
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 18) Roadmap (CLI)
|
||||
|
||||
* `scan fs <path>` (local filesystem tree) → upload to backend for analysis.
|
||||
* `policy test --sbom <file>` (simulate policy results offline using local policy bundle).
|
||||
* `runtime capture` (developer mode) — capture small `/proc/<pid>/maps` for troubleshooting.
|
||||
* Pluggable output renderers for SARIF/HTML (admin‑controlled).
|
||||
|
||||
---
|
||||
|
||||
## 19) Example CI snippets
|
||||
|
||||
**GitHub Actions (post‑build)**
|
||||
|
||||
```yaml
|
||||
- name: Login (device code w/ OIDC broker)
|
||||
run: stellaops auth login --json --authority ${{ secrets.AUTHORITY_URL }}
|
||||
|
||||
- name: Scan
|
||||
run: stellaops scan image ${{ steps.build.outputs.digest }} --wait --json
|
||||
|
||||
- name: Export (usage view, protobuf)
|
||||
run: stellaops export sbom ${{ steps.build.outputs.digest }} --view usage --format cdx-pb --out sbom.pb
|
||||
|
||||
- name: Verify attestation
|
||||
run: stellaops verify attestation --artifact $(sha256sum sbom.pb | cut -d' ' -f1) --json
|
||||
```
|
||||
|
||||
**GitLab (buildx generator)**
|
||||
|
||||
```yaml
|
||||
script:
|
||||
- stellaops buildx install
|
||||
- docker buildx build --attest=type=sbom,generator=stellaops/sbom-indexer -t $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA .
|
||||
- stellaops scan image $CI_REGISTRY_IMAGE@$IMAGE_DIGEST --wait --json
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 20) Test matrix (OS/arch)
|
||||
|
||||
* Linux: ubuntu‑20.04/22.04/24.04 (x64, arm64), alpine (musl).
|
||||
* macOS: 13–15 (x64, arm64).
|
||||
* Windows: 10/11, Server 2019/2022 (x64, arm64).
|
||||
* Docker engines: Docker Desktop, containerd‑based runners.
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -98,7 +98,7 @@ At startup, services **self‑advertise** their semver & channel; the UI surface
|
||||
**Gating policy**:
|
||||
|
||||
* **Core images** (Authority, Scanner, Concelier, Excititor, Attestor, UI): public **read**.
|
||||
* **Enterprise add‑ons** (if any) and **pre‑release**: private repos via the **Registry Token Service** (`src/StellaOps.Registry.TokenService`) which exchanges Authority-issued OpToks for short-lived Docker registry bearer tokens.
|
||||
* **Enterprise add‑ons** (if any) and **pre‑release**: private repos via the **Registry Token Service** (`src/Registry/StellaOps.Registry.TokenService`) which exchanges Authority-issued OpToks for short-lived Docker registry bearer tokens.
|
||||
|
||||
> Monetization lever is **signing** (PoE gate), not image pulls, so the core remains simple to consume.
|
||||
|
||||
|
||||
@@ -1,487 +1,487 @@
|
||||
# component_architecture_scanner.md — **Stella Ops Scanner** (2025Q4)
|
||||
|
||||
> **Scope.** Implementation‑ready architecture for the **Scanner** subsystem: WebService, Workers, analyzers, SBOM assembly (inventory & usage), per‑layer caching, three‑way diffs, artifact catalog (RustFS default + Mongo, S3-compatible fallback), attestation hand‑off, and scale/security posture. This document is the contract between the scanning plane and everything else (Policy, Excititor, Concelier, UI, CLI).
|
||||
|
||||
---
|
||||
|
||||
## 0) Mission & boundaries
|
||||
|
||||
**Mission.** Produce **deterministic**, **explainable** SBOMs and diffs for container images and filesystems, quickly and repeatedly, without guessing. Emit two views: **Inventory** (everything present) and **Usage** (entrypoint closure + actually linked libs). Attach attestations through **Signer→Attestor→Rekor v2**.
|
||||
|
||||
**Boundaries.**
|
||||
|
||||
* Scanner **does not** produce PASS/FAIL. The backend (Policy + Excititor + Concelier) decides presentation and verdicts.
|
||||
* Scanner **does not** keep third‑party SBOM warehouses. It may **bind** to existing attestations for exact hashes.
|
||||
* Core analyzers are **deterministic** (no fuzzy identity). Optional heuristic plug‑ins (e.g., patch‑presence) run under explicit flags and never contaminate the core SBOM.
|
||||
|
||||
---
|
||||
|
||||
## 1) Solution & project layout
|
||||
|
||||
```
|
||||
src/
|
||||
├─ StellaOps.Scanner.WebService/ # REST control plane, catalog, diff, exports
|
||||
├─ StellaOps.Scanner.Worker/ # queue consumer; executes analyzers
|
||||
├─ StellaOps.Scanner.Models/ # DTOs, evidence, graph nodes, CDX/SPDX adapters
|
||||
├─ StellaOps.Scanner.Storage/ # Mongo repositories; RustFS object client (default) + S3 fallback; ILM/GC
|
||||
├─ StellaOps.Scanner.Queue/ # queue abstraction (Redis/NATS/RabbitMQ)
|
||||
├─ StellaOps.Scanner.Cache/ # layer cache; file CAS; bloom/bitmap indexes
|
||||
├─ StellaOps.Scanner.EntryTrace/ # ENTRYPOINT/CMD → terminal program resolver (shell AST)
|
||||
├─ StellaOps.Scanner.Analyzers.OS.[Apk|Dpkg|Rpm]/
|
||||
├─ StellaOps.Scanner.Analyzers.Lang.[Java|Node|Python|Go|DotNet|Rust]/
|
||||
├─ StellaOps.Scanner.Analyzers.Native.[ELF|PE|MachO]/ # PE/Mach-O planned (M2)
|
||||
├─ StellaOps.Scanner.Emit.CDX/ # CycloneDX (JSON + Protobuf)
|
||||
├─ StellaOps.Scanner.Emit.SPDX/ # SPDX 3.0.1 JSON
|
||||
├─ StellaOps.Scanner.Diff/ # image→layer→component three‑way diff
|
||||
├─ StellaOps.Scanner.Index/ # BOM‑Index sidecar (purls + roaring bitmaps)
|
||||
├─ StellaOps.Scanner.Tests.* # unit/integration/e2e fixtures
|
||||
└─ tools/
|
||||
├─ StellaOps.Scanner.Sbomer.BuildXPlugin/ # BuildKit generator (image referrer SBOMs)
|
||||
└─ StellaOps.Scanner.Sbomer.DockerImage/ # CLI‑driven scanner container
|
||||
```
|
||||
|
||||
Analyzer assemblies and buildx generators are packaged as **restart-time plug-ins** under `plugins/scanner/**` with manifests; services must restart to activate new plug-ins.
|
||||
|
||||
### 1.1 Queue backbone (Redis / NATS)
|
||||
|
||||
`StellaOps.Scanner.Queue` exposes a transport-agnostic contract (`IScanQueue`/`IScanQueueLease`) used by the WebService producer and Worker consumers. Sprint 9 introduces two first-party transports:
|
||||
|
||||
- **Redis Streams** (default). Uses consumer groups, deterministic idempotency keys (`scanner:jobs:idemp:*`), and supports lease claim (`XCLAIM`), renewal, exponential-backoff retries, and a `scanner:jobs:dead` stream for exhausted attempts.
|
||||
- **NATS JetStream**. Provisions the `SCANNER_JOBS` work-queue stream + durable consumer `scanner-workers`, publishes with `MsgId` for dedupe, applies backoff via `NAK` delays, and routes dead-lettered jobs to `SCANNER_JOBS_DEAD`.
|
||||
|
||||
Metrics are emitted via `Meter` counters (`scanner_queue_enqueued_total`, `scanner_queue_retry_total`, `scanner_queue_deadletter_total`), and `ScannerQueueHealthCheck` pings the active backend (Redis `PING`, NATS `PING`). Configuration is bound from `scanner.queue`:
|
||||
|
||||
```yaml
|
||||
scanner:
|
||||
queue:
|
||||
kind: redis # or nats
|
||||
redis:
|
||||
connectionString: "redis://queue:6379/0"
|
||||
streamName: "scanner:jobs"
|
||||
nats:
|
||||
url: "nats://queue:4222"
|
||||
stream: "SCANNER_JOBS"
|
||||
subject: "scanner.jobs"
|
||||
durableConsumer: "scanner-workers"
|
||||
deadLetterSubject: "scanner.jobs.dead"
|
||||
maxDeliveryAttempts: 5
|
||||
retryInitialBackoff: 00:00:05
|
||||
retryMaxBackoff: 00:02:00
|
||||
```
|
||||
|
||||
The DI extension (`AddScannerQueue`) wires the selected transport, so future additions (e.g., RabbitMQ) only implement the same contract and register.
|
||||
|
||||
**Runtime form‑factor:** two deployables
|
||||
|
||||
* **Scanner.WebService** (stateless REST)
|
||||
* **Scanner.Worker** (N replicas; queue‑driven)
|
||||
|
||||
---
|
||||
|
||||
## 2) External dependencies
|
||||
|
||||
* **OCI registry** with **Referrers API** (discover attached SBOMs/signatures).
|
||||
* **RustFS** (default, offline-first) for SBOM artifacts; optional S3/MinIO compatibility retained for migration; **Object Lock** semantics emulated via retention headers; **ILM** for TTL.
|
||||
* **MongoDB** for catalog, job state, diffs, ILM rules.
|
||||
* **Queue** (Redis Streams/NATS/RabbitMQ).
|
||||
* **Authority** (on‑prem OIDC) for **OpToks** (DPoP/mTLS).
|
||||
* **Signer** + **Attestor** (+ **Fulcio/KMS** + **Rekor v2**) for DSSE + transparency.
|
||||
|
||||
---
|
||||
|
||||
## 3) Contracts & data model
|
||||
|
||||
### 3.1 Evidence‑first component model
|
||||
|
||||
**Nodes**
|
||||
|
||||
* `Image`, `Layer`, `File`
|
||||
* `Component` (`purl?`, `name`, `version?`, `type`, `id` — may be `bin:{sha256}`)
|
||||
* `Executable` (ELF/PE/Mach‑O), `Library` (native or managed), `EntryScript` (shell/launcher)
|
||||
|
||||
**Edges** (all carry **Evidence**)
|
||||
|
||||
* `contains(Image|Layer → File)`
|
||||
* `installs(PackageDB → Component)` (OS database row)
|
||||
* `declares(InstalledMetadata → Component)` (dist‑info, pom.properties, deps.json…)
|
||||
* `links_to(Executable → Library)` (ELF `DT_NEEDED`, PE imports)
|
||||
* `calls(EntryScript → Program)` (file:line from shell AST)
|
||||
* `attests(Rekor → Component|Image)` (SBOM/predicate binding)
|
||||
* `bound_from_attestation(Component_attested → Component_observed)` (hash equality proof)
|
||||
|
||||
**Evidence**
|
||||
|
||||
```
|
||||
{ source: enum, locator: (path|offset|line), sha256?, method: enum, timestamp }
|
||||
```
|
||||
|
||||
No confidences. Either a fact is proven with listed mechanisms, or it is not claimed.
|
||||
|
||||
### 3.2 Catalog schema (Mongo)
|
||||
|
||||
* `artifacts`
|
||||
|
||||
```
|
||||
{ _id, type: layer-bom|image-bom|diff|index,
|
||||
format: cdx-json|cdx-pb|spdx-json,
|
||||
bytesSha256, size, rekor: { uuid,index,url }?,
|
||||
ttlClass, immutable, refCount, createdAt }
|
||||
```
|
||||
* `images { imageDigest, repo, tag?, arch, createdAt, lastSeen }`
|
||||
* `layers { layerDigest, mediaType, size, createdAt, lastSeen }`
|
||||
* `links { fromType, fromDigest, artifactId }` // image/layer -> artifact
|
||||
* `jobs { _id, kind, args, state, startedAt, heartbeatAt, endedAt, error }`
|
||||
* `lifecycleRules { ruleId, scope, ttlDays, retainIfReferenced, immutable }`
|
||||
|
||||
### 3.3 Object store layout (RustFS)
|
||||
|
||||
```
|
||||
layers/<sha256>/sbom.cdx.json.zst
|
||||
layers/<sha256>/sbom.spdx.json.zst
|
||||
images/<imgDigest>/inventory.cdx.pb # CycloneDX Protobuf
|
||||
images/<imgDigest>/usage.cdx.pb
|
||||
indexes/<imgDigest>/bom-index.bin # purls + roaring bitmaps
|
||||
diffs/<old>_<new>/diff.json.zst
|
||||
attest/<artifactSha256>.dsse.json # DSSE bundle (cert chain + Rekor proof)
|
||||
```
|
||||
|
||||
RustFS exposes a deterministic HTTP API (`PUT|GET|DELETE /api/v1/buckets/{bucket}/objects/{key}`).
|
||||
Scanner clients tag immutable uploads with `X-RustFS-Immutable: true` and, when retention applies,
|
||||
`X-RustFS-Retain-Seconds: <ttlSeconds>`. Additional headers can be injected via
|
||||
`scanner.artifactStore.headers` to support custom auth or proxy requirements. Legacy MinIO/S3
|
||||
deployments remain supported by setting `scanner.artifactStore.driver = "s3"` during phased
|
||||
migrations.
|
||||
|
||||
---
|
||||
|
||||
## 4) REST API (Scanner.WebService)
|
||||
|
||||
All under `/api/v1/scanner`. Auth: **OpTok** (DPoP/mTLS); RBAC scopes.
|
||||
|
||||
```
|
||||
POST /scans { imageRef|digest, force?:bool } → { scanId }
|
||||
GET /scans/{id} → { status, imageDigest, artifacts[], rekor? }
|
||||
GET /sboms/{imageDigest} ?format=cdx-json|cdx-pb|spdx-json&view=inventory|usage → bytes
|
||||
GET /diff?old=<digest>&new=<digest>&view=inventory|usage → diff.json
|
||||
POST /exports { imageDigest, format, view, attest?:bool } → { artifactId, rekor? }
|
||||
POST /reports { imageDigest, policyRevision? } → { reportId, rekor? } # delegates to backend policy+vex
|
||||
GET /catalog/artifacts/{id} → { meta }
|
||||
GET /healthz | /readyz | /metrics
|
||||
```
|
||||
|
||||
### Report events
|
||||
|
||||
When `scanner.events.enabled = true`, the WebService serialises the signed report (canonical JSON + DSSE envelope) with `NotifyCanonicalJsonSerializer` and publishes two Redis Stream entries (`scanner.report.ready`, `scanner.scan.completed`) to the configured stream (default `stella.events`). The stream fields carry the whole envelope plus lightweight headers (`kind`, `tenant`, `ts`) so Notify and UI timelines can consume the event bus without recomputing signatures. Publish timeouts and bounded stream length are controlled via `scanner:events:publishTimeoutSeconds` and `scanner:events:maxStreamLength`. If the queue driver is already Redis and no explicit events DSN is provided, the host reuses the queue connection and auto-enables event emission so deployments get live envelopes without extra wiring. Compose/Helm bundles expose the same knobs via the `SCANNER__EVENTS__*` environment variables for quick tuning.
|
||||
|
||||
---
|
||||
|
||||
## 5) Execution flow (Worker)
|
||||
|
||||
### 5.1 Acquire & verify
|
||||
|
||||
1. **Resolve image** (prefer `repo@sha256:…`).
|
||||
2. **(Optional) verify image signature** per policy (cosign).
|
||||
3. **Pull blobs**, compute layer digests; record metadata.
|
||||
|
||||
### 5.2 Layer union FS
|
||||
|
||||
* Apply whiteouts; materialize final filesystem; map **file → first introducing layer**.
|
||||
* Windows layers (MSI/SxS/GAC) planned in **M2**.
|
||||
|
||||
### 5.3 Evidence harvest (parallel analyzers; deterministic only)
|
||||
|
||||
**A) OS packages**
|
||||
|
||||
* **apk**: `/lib/apk/db/installed`
|
||||
* **dpkg**: `/var/lib/dpkg/status`, `/var/lib/dpkg/info/*.list`
|
||||
* **rpm**: `/var/lib/rpm/Packages` (via librpm or parser)
|
||||
* Record `name`, `version` (epoch/revision), `arch`, source package where present, and **declared file lists**.
|
||||
|
||||
> **Data flow note:** Each OS analyzer now writes its canonical output into the shared `ScanAnalysisStore` under
|
||||
> `analysis.os.packages` (raw results), `analysis.os.fragments` (per-analyzer layer fragments), and contributes to
|
||||
> `analysis.layers.fragments` (the aggregated view consumed by emit/diff pipelines). Helpers in
|
||||
> `ScanAnalysisCompositionBuilder` convert these fragments into SBOM composition requests and component graphs so the
|
||||
> diff/emit stages no longer reach back into individual analyzer implementations.
|
||||
|
||||
**B) Language ecosystems (installed state only)**
|
||||
|
||||
* **Java**: `META-INF/maven/*/pom.properties`, MANIFEST → `pkg:maven/...`
|
||||
* **Node**: `node_modules/**/package.json` → `pkg:npm/...`
|
||||
* **Python**: `*.dist-info/{METADATA,RECORD}` → `pkg:pypi/...`
|
||||
* **Go**: Go **buildinfo** in binaries → `pkg:golang/...`
|
||||
* **.NET**: `*.deps.json` + assembly metadata → `pkg:nuget/...`
|
||||
* **Rust**: crates only when **explicitly present** (embedded metadata or cargo/registry traces); otherwise binaries reported as `bin:{sha256}`.
|
||||
|
||||
> **Rule:** We only report components proven **on disk** with authoritative metadata. Lockfiles are evidence only.
|
||||
|
||||
**C) Native link graph**
|
||||
|
||||
* **ELF**: parse `PT_INTERP`, `DT_NEEDED`, RPATH/RUNPATH, **GNU symbol versions**; map **SONAMEs** to file paths; link executables → libs.
|
||||
* **PE/Mach‑O** (planned M2): import table, delay‑imports; version resources; code signatures.
|
||||
* Map libs back to **OS packages** if possible (via file lists); else emit `bin:{sha256}` components.
|
||||
* The exported metadata (`stellaops.os.*` properties, license list, source package) feeds policy scoring and export pipelines
|
||||
directly – Policy evaluates quiet rules against package provenance while Exporters forward the enriched fields into
|
||||
downstream JSON/Trivy payloads.
|
||||
|
||||
**D) EntryTrace (ENTRYPOINT/CMD → terminal program)**
|
||||
|
||||
* Read image config; parse shell (POSIX/Bash subset) with AST: `source`/`.` includes; `case/if`; `exec`/`command`; `run‑parts`.
|
||||
* Resolve commands via **PATH** within the **built rootfs**; follow language launchers (Java/Node/Python) to identify the terminal program (ELF/JAR/venv script).
|
||||
* Record **file:line** and choices for each hop; output chain graph.
|
||||
* Unresolvable dynamic constructs are recorded as **unknown** edges with reasons (e.g., `$FOO` unresolved).
|
||||
|
||||
**E) Attestation & SBOM bind (optional)**
|
||||
|
||||
* For each **file hash** or **binary hash**, query local cache of **Rekor v2** indices; if an SBOM attestation is found for **exact hash**, bind it to the component (origin=`attested`).
|
||||
* For the **image** digest, likewise bind SBOM attestations (build‑time referrers).
|
||||
|
||||
### 5.4 Component normalization (exact only)
|
||||
|
||||
* Create `Component` nodes only with deterministic identities: purl, or **`bin:{sha256}`** for unlabeled binaries.
|
||||
* Record **origin** (OS DB, installed metadata, linker, attestation).
|
||||
|
||||
### 5.5 SBOM assembly & emit
|
||||
|
||||
* **Per-layer SBOM fragments**: components introduced by the layer (+ relationships).
|
||||
* **Image SBOMs**: merge fragments; refer back to them via **CycloneDX BOM‑Link** (or SPDX ExternalRef).
|
||||
* Emit both **Inventory** & **Usage** views.
|
||||
* When the native analyzer reports an ELF `buildId`, attach it to component metadata and surface it as `stellaops:buildId` in CycloneDX properties (and diff metadata). This keeps SBOM/diff output in lockstep with runtime events and the debug-store manifest.
|
||||
* Serialize **CycloneDX JSON** and **CycloneDX Protobuf**; optionally **SPDX 3.0.1 JSON**.
|
||||
* Build **BOM‑Index** sidecar: purl table + roaring bitmap; flag `usedByEntrypoint` components for fast backend joins.
|
||||
|
||||
The emitted `buildId` metadata is preserved in component hashes, diff payloads, and `/policy/runtime` responses so operators can pivot from SBOM entries → runtime events → `debug/.build-id/<aa>/<rest>.debug` within the Offline Kit or release bundle.
|
||||
|
||||
### 5.6 DSSE attestation (via Signer/Attestor)
|
||||
|
||||
* WebService constructs **predicate** with `image_digest`, `stellaops_version`, `license_id`, `policy_digest?` (when emitting **final reports**), timestamps.
|
||||
* Calls **Signer** (requires **OpTok + PoE**); Signer verifies **entitlement + scanner image integrity** and returns **DSSE bundle**.
|
||||
* **Attestor** logs to **Rekor v2**; returns `{uuid,index,proof}` → stored in `artifacts.rekor`.
|
||||
|
||||
---
|
||||
|
||||
## 6) Three‑way diff (image → layer → component)
|
||||
|
||||
### 6.1 Keys & classification
|
||||
|
||||
* Component key: **purl** when present; else `bin:{sha256}`.
|
||||
* Diff classes: `added`, `removed`, `version_changed` (`upgraded|downgraded`), `metadata_changed` (e.g., origin from attestation vs observed).
|
||||
* Layer attribution: for each change, resolve the **introducing/removing layer**.
|
||||
|
||||
### 6.2 Algorithm (outline)
|
||||
|
||||
```
|
||||
A = components(imageOld, key)
|
||||
B = components(imageNew, key)
|
||||
|
||||
added = B \ A
|
||||
removed = A \ B
|
||||
changed = { k in A∩B : version(A[k]) != version(B[k]) || origin changed }
|
||||
|
||||
for each item in added/removed/changed:
|
||||
layer = attribute_to_layer(item, imageOld|imageNew)
|
||||
usageFlag = usedByEntrypoint(item, imageNew)
|
||||
emit diff.json (grouped by layer with badges)
|
||||
```
|
||||
|
||||
Diffs are stored as artifacts and feed **UI** and **CLI**.
|
||||
|
||||
---
|
||||
|
||||
## 7) Build‑time SBOMs (fast CI path)
|
||||
|
||||
**Scanner.Sbomer.BuildXPlugin** can act as a BuildKit **generator**:
|
||||
|
||||
* During `docker buildx build --attest=type=sbom,generator=stellaops/sbom-indexer`, run analyzers on the build context/output; attach SBOMs as OCI **referrers** to the built image.
|
||||
* Optionally request **Signer/Attestor** to produce **Stella Ops‑verified** attestation immediately; else, Scanner.WebService can verify and re‑attest post‑push.
|
||||
* Scanner.WebService trusts build‑time SBOMs per policy, enabling **no‑rescan** for unchanged bases.
|
||||
|
||||
---
|
||||
|
||||
## 8) Configuration (YAML)
|
||||
|
||||
```yaml
|
||||
scanner:
|
||||
queue:
|
||||
kind: redis
|
||||
url: "redis://queue:6379/0"
|
||||
mongo:
|
||||
uri: "mongodb://mongo/scanner"
|
||||
s3:
|
||||
endpoint: "http://minio:9000"
|
||||
bucket: "stellaops"
|
||||
objectLock: "governance" # or 'compliance'
|
||||
analyzers:
|
||||
os: { apk: true, dpkg: true, rpm: true }
|
||||
lang: { java: true, node: true, python: true, go: true, dotnet: true, rust: true }
|
||||
native: { elf: true, pe: false, macho: false } # PE/Mach-O in M2
|
||||
entryTrace: { enabled: true, shellMaxDepth: 64, followRunParts: true }
|
||||
emit:
|
||||
cdx: { json: true, protobuf: true }
|
||||
spdx: { json: true }
|
||||
compress: "zstd"
|
||||
rekor:
|
||||
url: "https://rekor-v2.internal"
|
||||
signer:
|
||||
url: "https://signer.internal"
|
||||
limits:
|
||||
maxParallel: 8
|
||||
perRegistryConcurrency: 2
|
||||
policyHints:
|
||||
verifyImageSignature: false
|
||||
trustBuildTimeSboms: true
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 9) Scale & performance
|
||||
|
||||
* **Parallelism**: per‑analyzer concurrency; bounded directory walkers; file CAS dedupe by sha256.
|
||||
* **Distributed locks** per **layer digest** to prevent duplicate work across Workers.
|
||||
* **Registry throttles**: per‑host concurrency budgets; exponential backoff on 429/5xx.
|
||||
* **Targets**:
|
||||
|
||||
* **Build‑time**: P95 ≤ 3–5 s on warmed bases (CI generator).
|
||||
* **Post‑build delta**: P95 ≤ 10 s for 200 MB images with cache hit.
|
||||
* **Emit**: CycloneDX Protobuf ≤ 150 ms for 5k components; JSON ≤ 500 ms.
|
||||
* **Diff**: ≤ 200 ms for 5k vs 5k components.
|
||||
|
||||
---
|
||||
|
||||
## 10) Security posture
|
||||
|
||||
* **AuthN**: Authority‑issued short OpToks (DPoP/mTLS).
|
||||
* **AuthZ**: scopes (`scanner.scan`, `scanner.export`, `scanner.catalog.read`).
|
||||
* **mTLS** to **Signer**/**Attestor**; only **Signer** can sign.
|
||||
* **No network fetches** during analysis (except registry pulls and optional Rekor index reads).
|
||||
* **Sandboxing**: non‑root containers; read‑only FS; seccomp profiles; disable execution of scanned content.
|
||||
* **Release integrity**: all first‑party images are **cosign‑signed**; Workers/WebService self‑verify at startup.
|
||||
|
||||
---
|
||||
|
||||
## 11) Observability & audit
|
||||
|
||||
* **Metrics**:
|
||||
|
||||
* `scanner.jobs_inflight`, `scanner.scan_latency_seconds`
|
||||
* `scanner.layer_cache_hits_total`, `scanner.file_cas_hits_total`
|
||||
* `scanner.artifact_bytes_total{format}`
|
||||
* `scanner.attestation_latency_seconds`, `scanner.rekor_failures_total`
|
||||
* `scanner_analyzer_golang_heuristic_total{indicator,version_hint}` — increments whenever the Go analyzer falls back to heuristics (build-id or runtime markers). Grafana panel: `sum by (indicator) (rate(scanner_analyzer_golang_heuristic_total[5m]))`; alert when the rate is ≥ 1 for 15 minutes to highlight unexpected stripped binaries.
|
||||
* **Tracing**: spans for acquire→union→analyzers→compose→emit→sign→log.
|
||||
* **Audit logs**: DSSE requests log `license_id`, `image_digest`, `artifactSha256`, `policy_digest?`, Rekor UUID on success.
|
||||
|
||||
---
|
||||
|
||||
## 12) Testing matrix
|
||||
|
||||
* **Determinism:** given same image + analyzers → byte‑identical **CDX Protobuf**; JSON normalized.
|
||||
* **OS packages:** ground‑truth images per distro; compare to package DB.
|
||||
* **Lang ecosystems:** sample images per ecosystem (Java/Node/Python/Go/.NET/Rust) with installed metadata; negative tests w/ lockfile‑only.
|
||||
* **Native & EntryTrace:** ELF graph correctness; shell AST cases (includes, run‑parts, exec, case/if).
|
||||
* **Diff:** layer attribution against synthetic two‑image sequences.
|
||||
* **Performance:** cold vs warm cache; large `node_modules` and `site‑packages`.
|
||||
* **Security:** ensure no code execution from image; fuzz parser inputs; path traversal resistance on layer extract.
|
||||
|
||||
---
|
||||
|
||||
## 13) Failure modes & degradations
|
||||
|
||||
* **Missing OS DB** (files exist, DB removed): record **files**; do **not** fabricate package components; emit `bin:{sha256}` where unavoidable; flag in evidence.
|
||||
* **Unreadable metadata** (corrupt dist‑info): record file evidence; skip component creation; annotate.
|
||||
* **Dynamic shell constructs**: mark unresolved edges with reasons (env var unknown) and continue; **Usage** view may be partial.
|
||||
* **Registry rate limits**: honor backoff; queue job retries with jitter.
|
||||
* **Signer refusal** (license/plan/version): scan completes; artifact produced; **no attestation**; WebService marks result as **unverified**.
|
||||
|
||||
---
|
||||
|
||||
## 14) Optional plug‑ins (off by default)
|
||||
|
||||
* **Patch‑presence detector** (signature‑based backport checks). Reads curated function‑level signatures from advisories; inspects binaries for patched code snippets to lower false‑positives for backported fixes. Runs as a sidecar analyzer that **annotates** components; never overrides core identities.
|
||||
* **Runtime probes** (with Zastava): when allowed, compare **/proc/<pid>/maps** (DSOs actually loaded) with static **Usage** view for precision.
|
||||
|
||||
---
|
||||
|
||||
## 15) DevOps & operations
|
||||
|
||||
* **HA**: WebService horizontal scale; Workers autoscale by queue depth & CPU; distributed locks on layers.
|
||||
* **Retention**: ILM rules per artifact class (`short`, `default`, `compliance`); **Object Lock** for compliance artifacts (reports, signed SBOMs).
|
||||
* **Upgrades**: bump **cache schema** when analyzer outputs change; WebService triggers refresh of dependent artifacts.
|
||||
* **Backups**: Mongo (daily dumps); RustFS snapshots (filesystem-level rsync/ZFS) or S3 versioning when legacy driver enabled; Rekor v2 DB snapshots.
|
||||
|
||||
---
|
||||
|
||||
## 16) CLI & UI touch points
|
||||
|
||||
* **CLI**: `stellaops scan <ref>`, `stellaops diff --old --new`, `stellaops export`, `stellaops verify attestation <bundle|url>`.
|
||||
* **UI**: Scan detail shows **Inventory/Usage** toggles, **Diff by Layer**, **Attestation badge** (verified/unverified), Rekor link, and **EntryTrace** chain with file:line breadcrumbs.
|
||||
|
||||
---
|
||||
|
||||
## 17) Roadmap (Scanner)
|
||||
|
||||
* **M2**: Windows containers (MSI/SxS/GAC analyzers), PE/Mach‑O native analyzer, deeper Rust metadata.
|
||||
* **M2**: Buildx generator GA (certified external registries), cross‑registry trust policies.
|
||||
* **M3**: Patch‑presence plug‑in GA (opt‑in), cross‑image corpus clustering (evidence‑only; not identity).
|
||||
* **M3**: Advanced EntryTrace (POSIX shell features breadth, busybox detection).
|
||||
|
||||
---
|
||||
|
||||
### Appendix A — EntryTrace resolution (pseudo)
|
||||
|
||||
```csharp
|
||||
ResolveEntrypoint(ImageConfig cfg, RootFs fs):
|
||||
cmd = Normalize(cfg.ENTRYPOINT, cfg.CMD)
|
||||
stack = [ Script(cmd, path=FindOnPath(cmd[0], fs)) ]
|
||||
visited = set()
|
||||
|
||||
while stack not empty and depth < MAX:
|
||||
cur = stack.pop()
|
||||
if cur in visited: continue
|
||||
visited.add(cur)
|
||||
|
||||
if IsShellScript(cur.path):
|
||||
ast = ParseShell(cur.path)
|
||||
foreach directive in ast:
|
||||
if directive is Source include:
|
||||
p = ResolveInclude(include.path, cur.env, fs)
|
||||
stack.push(Script(p))
|
||||
if directive is Exec call:
|
||||
p = ResolveExec(call.argv[0], cur.env, fs)
|
||||
stack.push(Program(p, argv=call.argv))
|
||||
if directive is Interpreter (python -m / node / java -jar):
|
||||
term = ResolveInterpreterTarget(call, fs)
|
||||
stack.push(Program(term))
|
||||
else:
|
||||
return Terminal(cur.path)
|
||||
|
||||
return Unknown(reason)
|
||||
```
|
||||
|
||||
### Appendix A.1 — EntryTrace Explainability
|
||||
|
||||
EntryTrace emits structured diagnostics and metrics so operators can quickly understand why resolution succeeded or degraded:
|
||||
|
||||
| Reason | Description | Typical Mitigation |
|
||||
|--------|-------------|--------------------|
|
||||
| `CommandNotFound` | A command referenced in the script cannot be located in the layered root filesystem or `PATH`. | Ensure binaries exist in the image or extend `PATH` hints. |
|
||||
| `MissingFile` | `source`/`.`/`run-parts` targets are missing. | Bundle the script or guard the include. |
|
||||
| `DynamicEnvironmentReference` | Path depends on `$VARS` that are unknown at scan time. | Provide defaults via scan metadata or accept partial usage. |
|
||||
| `RecursionLimitReached` | Nested includes exceeded the analyzer depth limit (default 64). | Flatten indirection or increase the limit in options. |
|
||||
| `RunPartsEmpty` | `run-parts` directory contained no executable entries. | Remove empty directories or ignore if intentional. |
|
||||
| `JarNotFound` / `ModuleNotFound` | Java/Python targets missing, preventing interpreter tracing. | Ship the jar/module with the image or adjust the launcher. |
|
||||
|
||||
Diagnostics drive two metrics published by `EntryTraceMetrics`:
|
||||
|
||||
- `entrytrace_resolutions_total{outcome}` — resolution attempts segmented by outcome (`resolved`, `partiallyresolved`, `unresolved`).
|
||||
- `entrytrace_unresolved_total{reason}` — diagnostic counts keyed by reason.
|
||||
|
||||
Structured logs include `entrytrace.path`, `entrytrace.command`, `entrytrace.reason`, and `entrytrace.depth`, all correlated with scan/job IDs. Timestamps are normalized to UTC (microsecond precision) to keep DSSE attestations and UI traces explainable.
|
||||
|
||||
### Appendix B — BOM‑Index sidecar
|
||||
|
||||
```
|
||||
struct Header { magic, version, imageDigest, createdAt }
|
||||
vector<string> purls
|
||||
map<purlIndex, roaring_bitmap> components
|
||||
optional map<purlIndex, roaring_bitmap> usedByEntrypoint
|
||||
```
|
||||
# component_architecture_scanner.md — **Stella Ops Scanner** (2025Q4)
|
||||
|
||||
> **Scope.** Implementation‑ready architecture for the **Scanner** subsystem: WebService, Workers, analyzers, SBOM assembly (inventory & usage), per‑layer caching, three‑way diffs, artifact catalog (RustFS default + Mongo, S3-compatible fallback), attestation hand‑off, and scale/security posture. This document is the contract between the scanning plane and everything else (Policy, Excititor, Concelier, UI, CLI).
|
||||
|
||||
---
|
||||
|
||||
## 0) Mission & boundaries
|
||||
|
||||
**Mission.** Produce **deterministic**, **explainable** SBOMs and diffs for container images and filesystems, quickly and repeatedly, without guessing. Emit two views: **Inventory** (everything present) and **Usage** (entrypoint closure + actually linked libs). Attach attestations through **Signer→Attestor→Rekor v2**.
|
||||
|
||||
**Boundaries.**
|
||||
|
||||
* Scanner **does not** produce PASS/FAIL. The backend (Policy + Excititor + Concelier) decides presentation and verdicts.
|
||||
* Scanner **does not** keep third‑party SBOM warehouses. It may **bind** to existing attestations for exact hashes.
|
||||
* Core analyzers are **deterministic** (no fuzzy identity). Optional heuristic plug‑ins (e.g., patch‑presence) run under explicit flags and never contaminate the core SBOM.
|
||||
|
||||
---
|
||||
|
||||
## 1) Solution & project layout
|
||||
|
||||
```
|
||||
src/
|
||||
├─ StellaOps.Scanner.WebService/ # REST control plane, catalog, diff, exports
|
||||
├─ StellaOps.Scanner.Worker/ # queue consumer; executes analyzers
|
||||
├─ StellaOps.Scanner.Models/ # DTOs, evidence, graph nodes, CDX/SPDX adapters
|
||||
├─ StellaOps.Scanner.Storage/ # Mongo repositories; RustFS object client (default) + S3 fallback; ILM/GC
|
||||
├─ StellaOps.Scanner.Queue/ # queue abstraction (Redis/NATS/RabbitMQ)
|
||||
├─ StellaOps.Scanner.Cache/ # layer cache; file CAS; bloom/bitmap indexes
|
||||
├─ StellaOps.Scanner.EntryTrace/ # ENTRYPOINT/CMD → terminal program resolver (shell AST)
|
||||
├─ StellaOps.Scanner.Analyzers.OS.[Apk|Dpkg|Rpm]/
|
||||
├─ StellaOps.Scanner.Analyzers.Lang.[Java|Node|Python|Go|DotNet|Rust]/
|
||||
├─ StellaOps.Scanner.Analyzers.Native.[ELF|PE|MachO]/ # PE/Mach-O planned (M2)
|
||||
├─ StellaOps.Scanner.Emit.CDX/ # CycloneDX (JSON + Protobuf)
|
||||
├─ StellaOps.Scanner.Emit.SPDX/ # SPDX 3.0.1 JSON
|
||||
├─ StellaOps.Scanner.Diff/ # image→layer→component three‑way diff
|
||||
├─ StellaOps.Scanner.Index/ # BOM‑Index sidecar (purls + roaring bitmaps)
|
||||
├─ StellaOps.Scanner.Tests.* # unit/integration/e2e fixtures
|
||||
└─ tools/
|
||||
├─ StellaOps.Scanner.Sbomer.BuildXPlugin/ # BuildKit generator (image referrer SBOMs)
|
||||
└─ StellaOps.Scanner.Sbomer.DockerImage/ # CLI‑driven scanner container
|
||||
```
|
||||
|
||||
Analyzer assemblies and buildx generators are packaged as **restart-time plug-ins** under `plugins/scanner/**` with manifests; services must restart to activate new plug-ins.
|
||||
|
||||
### 1.1 Queue backbone (Redis / NATS)
|
||||
|
||||
`StellaOps.Scanner.Queue` exposes a transport-agnostic contract (`IScanQueue`/`IScanQueueLease`) used by the WebService producer and Worker consumers. Sprint 9 introduces two first-party transports:
|
||||
|
||||
- **Redis Streams** (default). Uses consumer groups, deterministic idempotency keys (`scanner:jobs:idemp:*`), and supports lease claim (`XCLAIM`), renewal, exponential-backoff retries, and a `scanner:jobs:dead` stream for exhausted attempts.
|
||||
- **NATS JetStream**. Provisions the `SCANNER_JOBS` work-queue stream + durable consumer `scanner-workers`, publishes with `MsgId` for dedupe, applies backoff via `NAK` delays, and routes dead-lettered jobs to `SCANNER_JOBS_DEAD`.
|
||||
|
||||
Metrics are emitted via `Meter` counters (`scanner_queue_enqueued_total`, `scanner_queue_retry_total`, `scanner_queue_deadletter_total`), and `ScannerQueueHealthCheck` pings the active backend (Redis `PING`, NATS `PING`). Configuration is bound from `scanner.queue`:
|
||||
|
||||
```yaml
|
||||
scanner:
|
||||
queue:
|
||||
kind: redis # or nats
|
||||
redis:
|
||||
connectionString: "redis://queue:6379/0"
|
||||
streamName: "scanner:jobs"
|
||||
nats:
|
||||
url: "nats://queue:4222"
|
||||
stream: "SCANNER_JOBS"
|
||||
subject: "scanner.jobs"
|
||||
durableConsumer: "scanner-workers"
|
||||
deadLetterSubject: "scanner.jobs.dead"
|
||||
maxDeliveryAttempts: 5
|
||||
retryInitialBackoff: 00:00:05
|
||||
retryMaxBackoff: 00:02:00
|
||||
```
|
||||
|
||||
The DI extension (`AddScannerQueue`) wires the selected transport, so future additions (e.g., RabbitMQ) only implement the same contract and register.
|
||||
|
||||
**Runtime form‑factor:** two deployables
|
||||
|
||||
* **Scanner.WebService** (stateless REST)
|
||||
* **Scanner.Worker** (N replicas; queue‑driven)
|
||||
|
||||
---
|
||||
|
||||
## 2) External dependencies
|
||||
|
||||
* **OCI registry** with **Referrers API** (discover attached SBOMs/signatures).
|
||||
* **RustFS** (default, offline-first) for SBOM artifacts; optional S3/MinIO compatibility retained for migration; **Object Lock** semantics emulated via retention headers; **ILM** for TTL.
|
||||
* **MongoDB** for catalog, job state, diffs, ILM rules.
|
||||
* **Queue** (Redis Streams/NATS/RabbitMQ).
|
||||
* **Authority** (on‑prem OIDC) for **OpToks** (DPoP/mTLS).
|
||||
* **Signer** + **Attestor** (+ **Fulcio/KMS** + **Rekor v2**) for DSSE + transparency.
|
||||
|
||||
---
|
||||
|
||||
## 3) Contracts & data model
|
||||
|
||||
### 3.1 Evidence‑first component model
|
||||
|
||||
**Nodes**
|
||||
|
||||
* `Image`, `Layer`, `File`
|
||||
* `Component` (`purl?`, `name`, `version?`, `type`, `id` — may be `bin:{sha256}`)
|
||||
* `Executable` (ELF/PE/Mach‑O), `Library` (native or managed), `EntryScript` (shell/launcher)
|
||||
|
||||
**Edges** (all carry **Evidence**)
|
||||
|
||||
* `contains(Image|Layer → File)`
|
||||
* `installs(PackageDB → Component)` (OS database row)
|
||||
* `declares(InstalledMetadata → Component)` (dist‑info, pom.properties, deps.json…)
|
||||
* `links_to(Executable → Library)` (ELF `DT_NEEDED`, PE imports)
|
||||
* `calls(EntryScript → Program)` (file:line from shell AST)
|
||||
* `attests(Rekor → Component|Image)` (SBOM/predicate binding)
|
||||
* `bound_from_attestation(Component_attested → Component_observed)` (hash equality proof)
|
||||
|
||||
**Evidence**
|
||||
|
||||
```
|
||||
{ source: enum, locator: (path|offset|line), sha256?, method: enum, timestamp }
|
||||
```
|
||||
|
||||
No confidences. Either a fact is proven with listed mechanisms, or it is not claimed.
|
||||
|
||||
### 3.2 Catalog schema (Mongo)
|
||||
|
||||
* `artifacts`
|
||||
|
||||
```
|
||||
{ _id, type: layer-bom|image-bom|diff|index,
|
||||
format: cdx-json|cdx-pb|spdx-json,
|
||||
bytesSha256, size, rekor: { uuid,index,url }?,
|
||||
ttlClass, immutable, refCount, createdAt }
|
||||
```
|
||||
* `images { imageDigest, repo, tag?, arch, createdAt, lastSeen }`
|
||||
* `layers { layerDigest, mediaType, size, createdAt, lastSeen }`
|
||||
* `links { fromType, fromDigest, artifactId }` // image/layer -> artifact
|
||||
* `jobs { _id, kind, args, state, startedAt, heartbeatAt, endedAt, error }`
|
||||
* `lifecycleRules { ruleId, scope, ttlDays, retainIfReferenced, immutable }`
|
||||
|
||||
### 3.3 Object store layout (RustFS)
|
||||
|
||||
```
|
||||
layers/<sha256>/sbom.cdx.json.zst
|
||||
layers/<sha256>/sbom.spdx.json.zst
|
||||
images/<imgDigest>/inventory.cdx.pb # CycloneDX Protobuf
|
||||
images/<imgDigest>/usage.cdx.pb
|
||||
indexes/<imgDigest>/bom-index.bin # purls + roaring bitmaps
|
||||
diffs/<old>_<new>/diff.json.zst
|
||||
attest/<artifactSha256>.dsse.json # DSSE bundle (cert chain + Rekor proof)
|
||||
```
|
||||
|
||||
RustFS exposes a deterministic HTTP API (`PUT|GET|DELETE /api/v1/buckets/{bucket}/objects/{key}`).
|
||||
Scanner clients tag immutable uploads with `X-RustFS-Immutable: true` and, when retention applies,
|
||||
`X-RustFS-Retain-Seconds: <ttlSeconds>`. Additional headers can be injected via
|
||||
`scanner.artifactStore.headers` to support custom auth or proxy requirements. Legacy MinIO/S3
|
||||
deployments remain supported by setting `scanner.artifactStore.driver = "s3"` during phased
|
||||
migrations.
|
||||
|
||||
---
|
||||
|
||||
## 4) REST API (Scanner.WebService)
|
||||
|
||||
All under `/api/v1/scanner`. Auth: **OpTok** (DPoP/mTLS); RBAC scopes.
|
||||
|
||||
```
|
||||
POST /scans { imageRef|digest, force?:bool } → { scanId }
|
||||
GET /scans/{id} → { status, imageDigest, artifacts[], rekor? }
|
||||
GET /sboms/{imageDigest} ?format=cdx-json|cdx-pb|spdx-json&view=inventory|usage → bytes
|
||||
GET /diff?old=<digest>&new=<digest>&view=inventory|usage → diff.json
|
||||
POST /exports { imageDigest, format, view, attest?:bool } → { artifactId, rekor? }
|
||||
POST /reports { imageDigest, policyRevision? } → { reportId, rekor? } # delegates to backend policy+vex
|
||||
GET /catalog/artifacts/{id} → { meta }
|
||||
GET /healthz | /readyz | /metrics
|
||||
```
|
||||
|
||||
### Report events
|
||||
|
||||
When `scanner.events.enabled = true`, the WebService serialises the signed report (canonical JSON + DSSE envelope) with `NotifyCanonicalJsonSerializer` and publishes two Redis Stream entries (`scanner.report.ready`, `scanner.scan.completed`) to the configured stream (default `stella.events`). The stream fields carry the whole envelope plus lightweight headers (`kind`, `tenant`, `ts`) so Notify and UI timelines can consume the event bus without recomputing signatures. Publish timeouts and bounded stream length are controlled via `scanner:events:publishTimeoutSeconds` and `scanner:events:maxStreamLength`. If the queue driver is already Redis and no explicit events DSN is provided, the host reuses the queue connection and auto-enables event emission so deployments get live envelopes without extra wiring. Compose/Helm bundles expose the same knobs via the `SCANNER__EVENTS__*` environment variables for quick tuning.
|
||||
|
||||
---
|
||||
|
||||
## 5) Execution flow (Worker)
|
||||
|
||||
### 5.1 Acquire & verify
|
||||
|
||||
1. **Resolve image** (prefer `repo@sha256:…`).
|
||||
2. **(Optional) verify image signature** per policy (cosign).
|
||||
3. **Pull blobs**, compute layer digests; record metadata.
|
||||
|
||||
### 5.2 Layer union FS
|
||||
|
||||
* Apply whiteouts; materialize final filesystem; map **file → first introducing layer**.
|
||||
* Windows layers (MSI/SxS/GAC) planned in **M2**.
|
||||
|
||||
### 5.3 Evidence harvest (parallel analyzers; deterministic only)
|
||||
|
||||
**A) OS packages**
|
||||
|
||||
* **apk**: `/lib/apk/db/installed`
|
||||
* **dpkg**: `/var/lib/dpkg/status`, `/var/lib/dpkg/info/*.list`
|
||||
* **rpm**: `/var/lib/rpm/Packages` (via librpm or parser)
|
||||
* Record `name`, `version` (epoch/revision), `arch`, source package where present, and **declared file lists**.
|
||||
|
||||
> **Data flow note:** Each OS analyzer now writes its canonical output into the shared `ScanAnalysisStore` under
|
||||
> `analysis.os.packages` (raw results), `analysis.os.fragments` (per-analyzer layer fragments), and contributes to
|
||||
> `analysis.layers.fragments` (the aggregated view consumed by emit/diff pipelines). Helpers in
|
||||
> `ScanAnalysisCompositionBuilder` convert these fragments into SBOM composition requests and component graphs so the
|
||||
> diff/emit stages no longer reach back into individual analyzer implementations.
|
||||
|
||||
**B) Language ecosystems (installed state only)**
|
||||
|
||||
* **Java**: `META-INF/maven/*/pom.properties`, MANIFEST → `pkg:maven/...`
|
||||
* **Node**: `node_modules/**/package.json` → `pkg:npm/...`
|
||||
* **Python**: `*.dist-info/{METADATA,RECORD}` → `pkg:pypi/...`
|
||||
* **Go**: Go **buildinfo** in binaries → `pkg:golang/...`
|
||||
* **.NET**: `*.deps.json` + assembly metadata → `pkg:nuget/...`
|
||||
* **Rust**: crates only when **explicitly present** (embedded metadata or cargo/registry traces); otherwise binaries reported as `bin:{sha256}`.
|
||||
|
||||
> **Rule:** We only report components proven **on disk** with authoritative metadata. Lockfiles are evidence only.
|
||||
|
||||
**C) Native link graph**
|
||||
|
||||
* **ELF**: parse `PT_INTERP`, `DT_NEEDED`, RPATH/RUNPATH, **GNU symbol versions**; map **SONAMEs** to file paths; link executables → libs.
|
||||
* **PE/Mach‑O** (planned M2): import table, delay‑imports; version resources; code signatures.
|
||||
* Map libs back to **OS packages** if possible (via file lists); else emit `bin:{sha256}` components.
|
||||
* The exported metadata (`stellaops.os.*` properties, license list, source package) feeds policy scoring and export pipelines
|
||||
directly – Policy evaluates quiet rules against package provenance while Exporters forward the enriched fields into
|
||||
downstream JSON/Trivy payloads.
|
||||
|
||||
**D) EntryTrace (ENTRYPOINT/CMD → terminal program)**
|
||||
|
||||
* Read image config; parse shell (POSIX/Bash subset) with AST: `source`/`.` includes; `case/if`; `exec`/`command`; `run‑parts`.
|
||||
* Resolve commands via **PATH** within the **built rootfs**; follow language launchers (Java/Node/Python) to identify the terminal program (ELF/JAR/venv script).
|
||||
* Record **file:line** and choices for each hop; output chain graph.
|
||||
* Unresolvable dynamic constructs are recorded as **unknown** edges with reasons (e.g., `$FOO` unresolved).
|
||||
|
||||
**E) Attestation & SBOM bind (optional)**
|
||||
|
||||
* For each **file hash** or **binary hash**, query local cache of **Rekor v2** indices; if an SBOM attestation is found for **exact hash**, bind it to the component (origin=`attested`).
|
||||
* For the **image** digest, likewise bind SBOM attestations (build‑time referrers).
|
||||
|
||||
### 5.4 Component normalization (exact only)
|
||||
|
||||
* Create `Component` nodes only with deterministic identities: purl, or **`bin:{sha256}`** for unlabeled binaries.
|
||||
* Record **origin** (OS DB, installed metadata, linker, attestation).
|
||||
|
||||
### 5.5 SBOM assembly & emit
|
||||
|
||||
* **Per-layer SBOM fragments**: components introduced by the layer (+ relationships).
|
||||
* **Image SBOMs**: merge fragments; refer back to them via **CycloneDX BOM‑Link** (or SPDX ExternalRef).
|
||||
* Emit both **Inventory** & **Usage** views.
|
||||
* When the native analyzer reports an ELF `buildId`, attach it to component metadata and surface it as `stellaops:buildId` in CycloneDX properties (and diff metadata). This keeps SBOM/diff output in lockstep with runtime events and the debug-store manifest.
|
||||
* Serialize **CycloneDX JSON** and **CycloneDX Protobuf**; optionally **SPDX 3.0.1 JSON**.
|
||||
* Build **BOM‑Index** sidecar: purl table + roaring bitmap; flag `usedByEntrypoint` components for fast backend joins.
|
||||
|
||||
The emitted `buildId` metadata is preserved in component hashes, diff payloads, and `/policy/runtime` responses so operators can pivot from SBOM entries → runtime events → `debug/.build-id/<aa>/<rest>.debug` within the Offline Kit or release bundle.
|
||||
|
||||
### 5.6 DSSE attestation (via Signer/Attestor)
|
||||
|
||||
* WebService constructs **predicate** with `image_digest`, `stellaops_version`, `license_id`, `policy_digest?` (when emitting **final reports**), timestamps.
|
||||
* Calls **Signer** (requires **OpTok + PoE**); Signer verifies **entitlement + scanner image integrity** and returns **DSSE bundle**.
|
||||
* **Attestor** logs to **Rekor v2**; returns `{uuid,index,proof}` → stored in `artifacts.rekor`.
|
||||
|
||||
---
|
||||
|
||||
## 6) Three‑way diff (image → layer → component)
|
||||
|
||||
### 6.1 Keys & classification
|
||||
|
||||
* Component key: **purl** when present; else `bin:{sha256}`.
|
||||
* Diff classes: `added`, `removed`, `version_changed` (`upgraded|downgraded`), `metadata_changed` (e.g., origin from attestation vs observed).
|
||||
* Layer attribution: for each change, resolve the **introducing/removing layer**.
|
||||
|
||||
### 6.2 Algorithm (outline)
|
||||
|
||||
```
|
||||
A = components(imageOld, key)
|
||||
B = components(imageNew, key)
|
||||
|
||||
added = B \ A
|
||||
removed = A \ B
|
||||
changed = { k in A∩B : version(A[k]) != version(B[k]) || origin changed }
|
||||
|
||||
for each item in added/removed/changed:
|
||||
layer = attribute_to_layer(item, imageOld|imageNew)
|
||||
usageFlag = usedByEntrypoint(item, imageNew)
|
||||
emit diff.json (grouped by layer with badges)
|
||||
```
|
||||
|
||||
Diffs are stored as artifacts and feed **UI** and **CLI**.
|
||||
|
||||
---
|
||||
|
||||
## 7) Build‑time SBOMs (fast CI path)
|
||||
|
||||
**Scanner.Sbomer.BuildXPlugin** can act as a BuildKit **generator**:
|
||||
|
||||
* During `docker buildx build --attest=type=sbom,generator=stellaops/sbom-indexer`, run analyzers on the build context/output; attach SBOMs as OCI **referrers** to the built image.
|
||||
* Optionally request **Signer/Attestor** to produce **Stella Ops‑verified** attestation immediately; else, Scanner.WebService can verify and re‑attest post‑push.
|
||||
* Scanner.WebService trusts build‑time SBOMs per policy, enabling **no‑rescan** for unchanged bases.
|
||||
|
||||
---
|
||||
|
||||
## 8) Configuration (YAML)
|
||||
|
||||
```yaml
|
||||
scanner:
|
||||
queue:
|
||||
kind: redis
|
||||
url: "redis://queue:6379/0"
|
||||
mongo:
|
||||
uri: "mongodb://mongo/scanner"
|
||||
s3:
|
||||
endpoint: "http://minio:9000"
|
||||
bucket: "stellaops"
|
||||
objectLock: "governance" # or 'compliance'
|
||||
analyzers:
|
||||
os: { apk: true, dpkg: true, rpm: true }
|
||||
lang: { java: true, node: true, python: true, go: true, dotnet: true, rust: true }
|
||||
native: { elf: true, pe: false, macho: false } # PE/Mach-O in M2
|
||||
entryTrace: { enabled: true, shellMaxDepth: 64, followRunParts: true }
|
||||
emit:
|
||||
cdx: { json: true, protobuf: true }
|
||||
spdx: { json: true }
|
||||
compress: "zstd"
|
||||
rekor:
|
||||
url: "https://rekor-v2.internal"
|
||||
signer:
|
||||
url: "https://signer.internal"
|
||||
limits:
|
||||
maxParallel: 8
|
||||
perRegistryConcurrency: 2
|
||||
policyHints:
|
||||
verifyImageSignature: false
|
||||
trustBuildTimeSboms: true
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 9) Scale & performance
|
||||
|
||||
* **Parallelism**: per‑analyzer concurrency; bounded directory walkers; file CAS dedupe by sha256.
|
||||
* **Distributed locks** per **layer digest** to prevent duplicate work across Workers.
|
||||
* **Registry throttles**: per‑host concurrency budgets; exponential backoff on 429/5xx.
|
||||
* **Targets**:
|
||||
|
||||
* **Build‑time**: P95 ≤ 3–5 s on warmed bases (CI generator).
|
||||
* **Post‑build delta**: P95 ≤ 10 s for 200 MB images with cache hit.
|
||||
* **Emit**: CycloneDX Protobuf ≤ 150 ms for 5k components; JSON ≤ 500 ms.
|
||||
* **Diff**: ≤ 200 ms for 5k vs 5k components.
|
||||
|
||||
---
|
||||
|
||||
## 10) Security posture
|
||||
|
||||
* **AuthN**: Authority‑issued short OpToks (DPoP/mTLS).
|
||||
* **AuthZ**: scopes (`scanner.scan`, `scanner.export`, `scanner.catalog.read`).
|
||||
* **mTLS** to **Signer**/**Attestor**; only **Signer** can sign.
|
||||
* **No network fetches** during analysis (except registry pulls and optional Rekor index reads).
|
||||
* **Sandboxing**: non‑root containers; read‑only FS; seccomp profiles; disable execution of scanned content.
|
||||
* **Release integrity**: all first‑party images are **cosign‑signed**; Workers/WebService self‑verify at startup.
|
||||
|
||||
---
|
||||
|
||||
## 11) Observability & audit
|
||||
|
||||
* **Metrics**:
|
||||
|
||||
* `scanner.jobs_inflight`, `scanner.scan_latency_seconds`
|
||||
* `scanner.layer_cache_hits_total`, `scanner.file_cas_hits_total`
|
||||
* `scanner.artifact_bytes_total{format}`
|
||||
* `scanner.attestation_latency_seconds`, `scanner.rekor_failures_total`
|
||||
* `scanner_analyzer_golang_heuristic_total{indicator,version_hint}` — increments whenever the Go analyzer falls back to heuristics (build-id or runtime markers). Grafana panel: `sum by (indicator) (rate(scanner_analyzer_golang_heuristic_total[5m]))`; alert when the rate is ≥ 1 for 15 minutes to highlight unexpected stripped binaries.
|
||||
* **Tracing**: spans for acquire→union→analyzers→compose→emit→sign→log.
|
||||
* **Audit logs**: DSSE requests log `license_id`, `image_digest`, `artifactSha256`, `policy_digest?`, Rekor UUID on success.
|
||||
|
||||
---
|
||||
|
||||
## 12) Testing matrix
|
||||
|
||||
* **Determinism:** given same image + analyzers → byte‑identical **CDX Protobuf**; JSON normalized.
|
||||
* **OS packages:** ground‑truth images per distro; compare to package DB.
|
||||
* **Lang ecosystems:** sample images per ecosystem (Java/Node/Python/Go/.NET/Rust) with installed metadata; negative tests w/ lockfile‑only.
|
||||
* **Native & EntryTrace:** ELF graph correctness; shell AST cases (includes, run‑parts, exec, case/if).
|
||||
* **Diff:** layer attribution against synthetic two‑image sequences.
|
||||
* **Performance:** cold vs warm cache; large `node_modules` and `site‑packages`.
|
||||
* **Security:** ensure no code execution from image; fuzz parser inputs; path traversal resistance on layer extract.
|
||||
|
||||
---
|
||||
|
||||
## 13) Failure modes & degradations
|
||||
|
||||
* **Missing OS DB** (files exist, DB removed): record **files**; do **not** fabricate package components; emit `bin:{sha256}` where unavoidable; flag in evidence.
|
||||
* **Unreadable metadata** (corrupt dist‑info): record file evidence; skip component creation; annotate.
|
||||
* **Dynamic shell constructs**: mark unresolved edges with reasons (env var unknown) and continue; **Usage** view may be partial.
|
||||
* **Registry rate limits**: honor backoff; queue job retries with jitter.
|
||||
* **Signer refusal** (license/plan/version): scan completes; artifact produced; **no attestation**; WebService marks result as **unverified**.
|
||||
|
||||
---
|
||||
|
||||
## 14) Optional plug‑ins (off by default)
|
||||
|
||||
* **Patch‑presence detector** (signature‑based backport checks). Reads curated function‑level signatures from advisories; inspects binaries for patched code snippets to lower false‑positives for backported fixes. Runs as a sidecar analyzer that **annotates** components; never overrides core identities.
|
||||
* **Runtime probes** (with Zastava): when allowed, compare **/proc/<pid>/maps** (DSOs actually loaded) with static **Usage** view for precision.
|
||||
|
||||
---
|
||||
|
||||
## 15) DevOps & operations
|
||||
|
||||
* **HA**: WebService horizontal scale; Workers autoscale by queue depth & CPU; distributed locks on layers.
|
||||
* **Retention**: ILM rules per artifact class (`short`, `default`, `compliance`); **Object Lock** for compliance artifacts (reports, signed SBOMs).
|
||||
* **Upgrades**: bump **cache schema** when analyzer outputs change; WebService triggers refresh of dependent artifacts.
|
||||
* **Backups**: Mongo (daily dumps); RustFS snapshots (filesystem-level rsync/ZFS) or S3 versioning when legacy driver enabled; Rekor v2 DB snapshots.
|
||||
|
||||
---
|
||||
|
||||
## 16) CLI & UI touch points
|
||||
|
||||
* **CLI**: `stellaops scan <ref>`, `stellaops diff --old --new`, `stellaops export`, `stellaops verify attestation <bundle|url>`.
|
||||
* **UI**: Scan detail shows **Inventory/Usage** toggles, **Diff by Layer**, **Attestation badge** (verified/unverified), Rekor link, and **EntryTrace** chain with file:line breadcrumbs.
|
||||
|
||||
---
|
||||
|
||||
## 17) Roadmap (Scanner)
|
||||
|
||||
* **M2**: Windows containers (MSI/SxS/GAC analyzers), PE/Mach‑O native analyzer, deeper Rust metadata.
|
||||
* **M2**: Buildx generator GA (certified external registries), cross‑registry trust policies.
|
||||
* **M3**: Patch‑presence plug‑in GA (opt‑in), cross‑image corpus clustering (evidence‑only; not identity).
|
||||
* **M3**: Advanced EntryTrace (POSIX shell features breadth, busybox detection).
|
||||
|
||||
---
|
||||
|
||||
### Appendix A — EntryTrace resolution (pseudo)
|
||||
|
||||
```csharp
|
||||
ResolveEntrypoint(ImageConfig cfg, RootFs fs):
|
||||
cmd = Normalize(cfg.ENTRYPOINT, cfg.CMD)
|
||||
stack = [ Script(cmd, path=FindOnPath(cmd[0], fs)) ]
|
||||
visited = set()
|
||||
|
||||
while stack not empty and depth < MAX:
|
||||
cur = stack.pop()
|
||||
if cur in visited: continue
|
||||
visited.add(cur)
|
||||
|
||||
if IsShellScript(cur.path):
|
||||
ast = ParseShell(cur.path)
|
||||
foreach directive in ast:
|
||||
if directive is Source include:
|
||||
p = ResolveInclude(include.path, cur.env, fs)
|
||||
stack.push(Script(p))
|
||||
if directive is Exec call:
|
||||
p = ResolveExec(call.argv[0], cur.env, fs)
|
||||
stack.push(Program(p, argv=call.argv))
|
||||
if directive is Interpreter (python -m / node / java -jar):
|
||||
term = ResolveInterpreterTarget(call, fs)
|
||||
stack.push(Program(term))
|
||||
else:
|
||||
return Terminal(cur.path)
|
||||
|
||||
return Unknown(reason)
|
||||
```
|
||||
|
||||
### Appendix A.1 — EntryTrace Explainability
|
||||
|
||||
EntryTrace emits structured diagnostics and metrics so operators can quickly understand why resolution succeeded or degraded:
|
||||
|
||||
| Reason | Description | Typical Mitigation |
|
||||
|--------|-------------|--------------------|
|
||||
| `CommandNotFound` | A command referenced in the script cannot be located in the layered root filesystem or `PATH`. | Ensure binaries exist in the image or extend `PATH` hints. |
|
||||
| `MissingFile` | `source`/`.`/`run-parts` targets are missing. | Bundle the script or guard the include. |
|
||||
| `DynamicEnvironmentReference` | Path depends on `$VARS` that are unknown at scan time. | Provide defaults via scan metadata or accept partial usage. |
|
||||
| `RecursionLimitReached` | Nested includes exceeded the analyzer depth limit (default 64). | Flatten indirection or increase the limit in options. |
|
||||
| `RunPartsEmpty` | `run-parts` directory contained no executable entries. | Remove empty directories or ignore if intentional. |
|
||||
| `JarNotFound` / `ModuleNotFound` | Java/Python targets missing, preventing interpreter tracing. | Ship the jar/module with the image or adjust the launcher. |
|
||||
|
||||
Diagnostics drive two metrics published by `EntryTraceMetrics`:
|
||||
|
||||
- `entrytrace_resolutions_total{outcome}` — resolution attempts segmented by outcome (`resolved`, `partiallyresolved`, `unresolved`).
|
||||
- `entrytrace_unresolved_total{reason}` — diagnostic counts keyed by reason.
|
||||
|
||||
Structured logs include `entrytrace.path`, `entrytrace.command`, `entrytrace.reason`, and `entrytrace.depth`, all correlated with scan/job IDs. Timestamps are normalized to UTC (microsecond precision) to keep DSSE attestations and UI traces explainable.
|
||||
|
||||
### Appendix B — BOM‑Index sidecar
|
||||
|
||||
```
|
||||
struct Header { magic, version, imageDigest, createdAt }
|
||||
vector<string> purls
|
||||
map<purlIndex, roaring_bitmap> components
|
||||
optional map<purlIndex, roaring_bitmap> usedByEntrypoint
|
||||
```
|
||||
|
||||
@@ -1,463 +1,463 @@
|
||||
# component_architecture_vexer.md — **Stella Ops Vexer** (2025Q4)
|
||||
|
||||
> **Scope.** This document specifies the **Vexer** service: its purpose, trust model, data structures, APIs, plug‑in contracts, storage schema, normalization/consensus algorithms, performance budgets, testing matrix, and how it integrates with Scanner, Policy, Feedser, and the attestation chain. It is implementation‑ready.
|
||||
|
||||
---
|
||||
|
||||
## 0) Mission & role in the platform
|
||||
|
||||
**Mission.** Convert heterogeneous **VEX** statements (OpenVEX, CSAF VEX, CycloneDX VEX; vendor/distro/platform sources) into **canonical, queryable claims**; compute **deterministic consensus** per *(vuln, product)*; preserve **conflicts with provenance**; publish **stable, attestable exports** that the backend uses to suppress non‑exploitable findings, prioritize remaining risk, and explain decisions.
|
||||
|
||||
**Boundaries.**
|
||||
|
||||
* Vexer **does not** decide PASS/FAIL. It supplies **evidence** (statuses + justifications + provenance weights).
|
||||
* Vexer preserves **conflicting claims** unchanged; consensus encodes how we would pick, but the raw set is always exportable.
|
||||
* VEX consumption is **backend‑only**: Scanner never applies VEX. The backend’s **Policy Engine** asks Vexer for status evidence and then decides what to show.
|
||||
|
||||
---
|
||||
|
||||
## 1) Inputs, outputs & canonical domain
|
||||
|
||||
### 1.1 Accepted input formats (ingest)
|
||||
|
||||
* **OpenVEX** JSON documents (attested or raw).
|
||||
* **CSAF VEX** 2.x (vendor PSIRTs and distros commonly publish CSAF).
|
||||
* **CycloneDX VEX** 1.4+ (standalone VEX or embedded VEX blocks).
|
||||
* **OCI‑attached attestations** (VEX statements shipped as OCI referrers) — optional connectors.
|
||||
|
||||
All connectors register **source metadata**: provider identity, trust tier, signature expectations (PGP/cosign/PKI), fetch windows, rate limits, and time anchors.
|
||||
|
||||
### 1.2 Canonical model (normalized)
|
||||
|
||||
Every incoming statement becomes a set of **VexClaim** records:
|
||||
|
||||
```
|
||||
VexClaim
|
||||
- providerId // 'redhat', 'suse', 'ubuntu', 'github', 'vendorX'
|
||||
- vulnId // 'CVE-2025-12345', 'GHSA-xxxx', canonicalized
|
||||
- productKey // canonical product identity (see §2.2)
|
||||
- status // affected | not_affected | fixed | under_investigation
|
||||
- justification? // for 'not_affected'/'affected' where provided
|
||||
- introducedVersion? // semantics per provider (range or exact)
|
||||
- fixedVersion? // where provided (range or exact)
|
||||
- lastObserved // timestamp from source or fetch time
|
||||
- provenance // doc digest, signature status, fetch URI, line/offset anchors
|
||||
- evidence[] // raw source snippets for explainability
|
||||
- supersedes? // optional cross-doc chain (docDigest → docDigest)
|
||||
```
|
||||
|
||||
### 1.3 Exports (consumption)
|
||||
|
||||
* **VexConsensus** per `(vulnId, productKey)` with:
|
||||
|
||||
* `rollupStatus` (after policy weights/justification gates),
|
||||
* `sources[]` (winning + losing claims with weights & reasons),
|
||||
* `policyRevisionId` (identifier of the Vexer policy used),
|
||||
* `consensusDigest` (stable SHA‑256 over canonical JSON).
|
||||
* **Raw claims** export for auditing (unchanged, with provenance).
|
||||
* **Provider snapshots** (per source, last N days) for operator debugging.
|
||||
* **Index** optimized for backend joins: `(productKey, vulnId) → (status, confidence, sourceSet)`.
|
||||
|
||||
All exports are **deterministic**, and (optionally) **attested** via DSSE and logged to Rekor v2.
|
||||
|
||||
---
|
||||
|
||||
## 2) Identity model — products & joins
|
||||
|
||||
### 2.1 Vuln identity
|
||||
|
||||
* Accepts **CVE**, **GHSA**, vendor IDs (MSRC, RHSA…), distro IDs (DSA/USN/RHSA…) — normalized to `vulnId` with alias sets.
|
||||
* **Alias graph** maintained (from Feedser) to map vendor/distro IDs → CVE (primary) and to **GHSA** where applicable.
|
||||
|
||||
### 2.2 Product identity (`productKey`)
|
||||
|
||||
* **Primary:** `purl` (Package URL).
|
||||
* **Secondary links:** `cpe`, **OS package NVRA/EVR**, NuGet/Maven/Golang identity, and **OS package name** when purl unavailable.
|
||||
* **Fallback:** `oci:<registry>/<repo>@<digest>` for image‑level VEX.
|
||||
* **Special cases:** kernel modules, firmware, platforms → provider‑specific mapping helpers (connector captures provider’s product taxonomy → canonical `productKey`).
|
||||
|
||||
> Vexer does not invent identities. If a provider cannot be mapped to purl/CPE/NVRA deterministically, we keep the native **product string** and mark the claim as **non‑joinable**; the backend will ignore it unless a policy explicitly whitelists that provider mapping.
|
||||
|
||||
---
|
||||
|
||||
## 3) Storage schema (MongoDB)
|
||||
|
||||
Database: `vexer`
|
||||
|
||||
### 3.1 Collections
|
||||
|
||||
**`vex.providers`**
|
||||
|
||||
```
|
||||
_id: providerId
|
||||
name, homepage, contact
|
||||
trustTier: enum {vendor, distro, platform, hub, attestation}
|
||||
signaturePolicy: { type: pgp|cosign|x509|none, keys[], certs[], cosignKeylessRoots[] }
|
||||
fetch: { baseUrl, kind: http|oci|file, rateLimit, etagSupport, windowDays }
|
||||
enabled: bool
|
||||
createdAt, modifiedAt
|
||||
```
|
||||
|
||||
**`vex.raw`** (immutable raw documents)
|
||||
|
||||
```
|
||||
_id: sha256(doc bytes)
|
||||
providerId
|
||||
uri
|
||||
ingestedAt
|
||||
contentType
|
||||
sig: { verified: bool, method: pgp|cosign|x509|none, keyId|certSubject, bundle? }
|
||||
payload: GridFS pointer (if large)
|
||||
disposition: kept|replaced|superseded
|
||||
correlation: { replaces?: sha256, replacedBy?: sha256 }
|
||||
```
|
||||
|
||||
**`vex.claims`** (normalized rows; dedupe on providerId+vulnId+productKey+docDigest)
|
||||
|
||||
```
|
||||
_id
|
||||
providerId
|
||||
vulnId
|
||||
productKey
|
||||
status
|
||||
justification?
|
||||
introducedVersion?
|
||||
fixedVersion?
|
||||
lastObserved
|
||||
docDigest
|
||||
provenance { uri, line?, pointer?, signatureState }
|
||||
evidence[] { key, value, locator }
|
||||
indices:
|
||||
- {vulnId:1, productKey:1}
|
||||
- {providerId:1, lastObserved:-1}
|
||||
- {status:1}
|
||||
- text index (optional) on evidence.value for debugging
|
||||
```
|
||||
|
||||
**`vex.consensus`** (rollups)
|
||||
|
||||
```
|
||||
_id: sha256(canonical(vulnId, productKey, policyRevision))
|
||||
vulnId
|
||||
productKey
|
||||
rollupStatus
|
||||
sources[]: [
|
||||
{ providerId, status, justification?, weight, lastObserved, accepted:bool, reason }
|
||||
]
|
||||
policyRevisionId
|
||||
evaluatedAt
|
||||
consensusDigest // same as _id
|
||||
indices:
|
||||
- {vulnId:1, productKey:1}
|
||||
- {policyRevisionId:1, evaluatedAt:-1}
|
||||
```
|
||||
|
||||
**`vex.exports`** (manifest of emitted artifacts)
|
||||
|
||||
```
|
||||
_id
|
||||
querySignature
|
||||
format: raw|consensus|index
|
||||
artifactSha256
|
||||
rekor { uuid, index, url }?
|
||||
createdAt
|
||||
policyRevisionId
|
||||
cacheable: bool
|
||||
```
|
||||
|
||||
**`vex.cache`**
|
||||
|
||||
```
|
||||
querySignature -> exportId (for fast reuse)
|
||||
ttl, hits
|
||||
```
|
||||
|
||||
**`vex.migrations`**
|
||||
|
||||
* ordered migrations applied at bootstrap to ensure indexes.
|
||||
|
||||
### 3.2 Indexing strategy
|
||||
|
||||
* Hot path queries use exact `(vulnId, productKey)` and time‑bounded windows; compound indexes cover both.
|
||||
* Providers list view by `lastObserved` for monitoring staleness.
|
||||
* `vex.consensus` keyed by `(vulnId, productKey, policyRevision)` for deterministic reuse.
|
||||
|
||||
---
|
||||
|
||||
## 4) Ingestion pipeline
|
||||
|
||||
### 4.1 Connector contract
|
||||
|
||||
```csharp
|
||||
public interface IVexConnector
|
||||
{
|
||||
string ProviderId { get; }
|
||||
Task FetchAsync(VexConnectorContext ctx, CancellationToken ct); // raw docs
|
||||
Task NormalizeAsync(VexConnectorContext ctx, CancellationToken ct); // raw -> VexClaim[]
|
||||
}
|
||||
```
|
||||
|
||||
* **Fetch** must implement: window scheduling, conditional GET (ETag/If‑Modified‑Since), rate limiting, retry/backoff.
|
||||
* **Normalize** parses the format, validates schema, maps product identities deterministically, emits `VexClaim` records with **provenance**.
|
||||
|
||||
### 4.2 Signature verification (per provider)
|
||||
|
||||
* **cosign (keyless or keyful)** for OCI referrers or HTTP‑served JSON with Sigstore bundles.
|
||||
* **PGP** (provider keyrings) for distro/vendor feeds that sign docs.
|
||||
* **x509** (mutual TLS / provider‑pinned certs) where applicable.
|
||||
* Signature state is stored on **vex.raw.sig** and copied into **provenance.signatureState** on claims.
|
||||
|
||||
> Claims from sources failing signature policy are marked `"signatureState.verified=false"` and **policy** can down‑weight or ignore them.
|
||||
|
||||
### 4.3 Time discipline
|
||||
|
||||
* For each doc, prefer **provider’s document timestamp**; if absent, use fetch time.
|
||||
* Claims carry `lastObserved` which drives **tie‑breaking** within equal weight tiers.
|
||||
|
||||
---
|
||||
|
||||
## 5) Normalization: product & status semantics
|
||||
|
||||
### 5.1 Product mapping
|
||||
|
||||
* **purl** first; **cpe** second; OS package NVRA/EVR mapping helpers (distro connectors) produce purls via canonical tables (e.g., rpm→purl:rpm, deb→purl:deb).
|
||||
* Where a provider publishes **platform‑level** VEX (e.g., “RHEL 9 not affected”), connectors expand to known product inventory rules (e.g., map to sets of packages/components shipped in the platform). Expansion tables are versioned and kept per provider; every expansion emits **evidence** indicating the rule applied.
|
||||
* If expansion would be speculative, the claim remains **platform‑scoped** with `productKey="platform:redhat:rhel:9"` and is flagged **non‑joinable**; backend can decide to use platform VEX only when Scanner proves the platform runtime.
|
||||
|
||||
### 5.2 Status + justification mapping
|
||||
|
||||
* Canonical **status**: `affected | not_affected | fixed | under_investigation`.
|
||||
* **Justifications** normalized to a controlled vocabulary (CISA‑aligned), e.g.:
|
||||
|
||||
* `component_not_present`
|
||||
* `vulnerable_code_not_in_execute_path`
|
||||
* `vulnerable_configuration_unused`
|
||||
* `inline_mitigation_applied`
|
||||
* `fix_available` (with `fixedVersion`)
|
||||
* `under_investigation`
|
||||
* Providers with free‑text justifications are mapped by deterministic tables; raw text preserved as `evidence`.
|
||||
|
||||
---
|
||||
|
||||
## 6) Consensus algorithm
|
||||
|
||||
**Goal:** produce a **stable**, explainable `rollupStatus` per `(vulnId, productKey)` given possibly conflicting claims.
|
||||
|
||||
### 6.1 Inputs
|
||||
|
||||
* Set **S** of `VexClaim` for the key.
|
||||
* **Vexer policy snapshot**:
|
||||
|
||||
* **weights** per provider tier and per provider overrides.
|
||||
* **justification gates** (e.g., require justification for `not_affected` to be acceptable).
|
||||
* **minEvidence** rules (e.g., `not_affected` must come from ≥1 vendor or 2 distros).
|
||||
* **signature requirements** (e.g., require verified signature for ‘fixed’ to be considered).
|
||||
|
||||
### 6.2 Steps
|
||||
|
||||
1. **Filter invalid** claims by signature policy & justification gates → set `S'`.
|
||||
2. **Score** each claim:
|
||||
`score = weight(provider) * freshnessFactor(lastObserved)` where freshnessFactor ∈ [0.8, 1.0] for staleness decay (configurable; small effect).
|
||||
3. **Aggregate** scores per status: `W(status) = Σ score(claims with that status)`.
|
||||
4. **Pick** `rollupStatus = argmax_status W(status)`.
|
||||
5. **Tie‑breakers** (in order):
|
||||
|
||||
* Higher **max single** provider score wins (vendor > distro > platform > hub).
|
||||
* More **recent** lastObserved wins.
|
||||
* Deterministic lexicographic order of status (`fixed` > `not_affected` > `under_investigation` > `affected`) as final tiebreaker.
|
||||
6. **Explain**: mark accepted sources (`accepted=true; reason="weight"`/`"freshness"`), mark rejected sources with explicit `reason` (`"insufficient_justification"`, `"signature_unverified"`, `"lower_weight"`).
|
||||
|
||||
> The algorithm is **pure** given S and policy snapshot; result is reproducible and hashed into `consensusDigest`.
|
||||
|
||||
---
|
||||
|
||||
## 7) Query & export APIs
|
||||
|
||||
All endpoints are versioned under `/api/v1/vex`.
|
||||
|
||||
### 7.1 Query (online)
|
||||
|
||||
```
|
||||
POST /claims/search
|
||||
body: { vulnIds?: string[], productKeys?: string[], providers?: string[], since?: timestamp, limit?: int, pageToken?: string }
|
||||
→ { claims[], nextPageToken? }
|
||||
|
||||
POST /consensus/search
|
||||
body: { vulnIds?: string[], productKeys?: string[], policyRevisionId?: string, since?: timestamp, limit?: int, pageToken?: string }
|
||||
→ { entries[], nextPageToken? }
|
||||
|
||||
POST /excititor/resolve (scope: vex.read)
|
||||
body: { productKeys?: string[], purls?: string[], vulnerabilityIds: string[], policyRevisionId?: string }
|
||||
→ { policy, resolvedAt, results: [ { vulnerabilityId, productKey, status, sources[], conflicts[], decisions[], signals?, summary?, envelope: { artifact, contentSignature?, attestation?, attestationEnvelope?, attestationSignature? } } ] }
|
||||
```
|
||||
|
||||
### 7.2 Exports (cacheable snapshots)
|
||||
|
||||
```
|
||||
POST /exports
|
||||
body: { signature: { vulnFilter?, productFilter?, providers?, since? }, format: raw|consensus|index, policyRevisionId?: string, force?: bool }
|
||||
→ { exportId, artifactSha256, rekor? }
|
||||
|
||||
GET /exports/{exportId} → bytes (application/json or binary index)
|
||||
GET /exports/{exportId}/meta → { signature, policyRevisionId, createdAt, artifactSha256, rekor? }
|
||||
```
|
||||
|
||||
### 7.3 Provider operations
|
||||
|
||||
```
|
||||
GET /providers → provider list & signature policy
|
||||
POST /providers/{id}/refresh → trigger fetch/normalize window
|
||||
GET /providers/{id}/status → last fetch, doc counts, signature stats
|
||||
```
|
||||
|
||||
**Auth:** service‑to‑service via Authority tokens; operator operations via UI/CLI with RBAC.
|
||||
|
||||
---
|
||||
|
||||
## 8) Attestation integration
|
||||
|
||||
* Exports can be **DSSE‑signed** via **Signer** and logged to **Rekor v2** via **Attestor** (optional but recommended for regulated pipelines).
|
||||
* `vex.exports.rekor` stores `{uuid, index, url}` when present.
|
||||
* **Predicate type**: `https://stella-ops.org/attestations/vex-export/1` with fields:
|
||||
|
||||
* `querySignature`, `policyRevisionId`, `artifactSha256`, `createdAt`.
|
||||
|
||||
---
|
||||
|
||||
## 9) Configuration (YAML)
|
||||
|
||||
```yaml
|
||||
vexer:
|
||||
mongo: { uri: "mongodb://mongo/vexer" }
|
||||
s3:
|
||||
endpoint: http://minio:9000
|
||||
bucket: stellaops
|
||||
policy:
|
||||
weights:
|
||||
vendor: 1.0
|
||||
distro: 0.9
|
||||
platform: 0.7
|
||||
hub: 0.5
|
||||
attestation: 0.6
|
||||
providerOverrides:
|
||||
redhat: 1.0
|
||||
suse: 0.95
|
||||
requireJustificationForNotAffected: true
|
||||
signatureRequiredForFixed: true
|
||||
minEvidence:
|
||||
not_affected:
|
||||
vendorOrTwoDistros: true
|
||||
connectors:
|
||||
- providerId: redhat
|
||||
kind: csaf
|
||||
baseUrl: https://access.redhat.com/security/data/csaf/v2/
|
||||
signaturePolicy: { type: pgp, keys: [ "…redhat-pgp-key…" ] }
|
||||
windowDays: 7
|
||||
- providerId: suse
|
||||
kind: csaf
|
||||
baseUrl: https://ftp.suse.com/pub/projects/security/csaf/
|
||||
signaturePolicy: { type: pgp, keys: [ "…suse-pgp-key…" ] }
|
||||
- providerId: ubuntu
|
||||
kind: openvex
|
||||
baseUrl: https://…/vex/
|
||||
signaturePolicy: { type: none }
|
||||
- providerId: vendorX
|
||||
kind: cyclonedx-vex
|
||||
ociRef: ghcr.io/vendorx/vex@sha256:…
|
||||
signaturePolicy: { type: cosign, cosignKeylessRoots: [ "sigstore-root" ] }
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 10) Security model
|
||||
|
||||
* **Input signature verification** enforced per provider policy (PGP, cosign, x509).
|
||||
* **Connector allowlists**: outbound fetch constrained to configured domains.
|
||||
* **Tenant isolation**: per‑tenant DB prefixes or separate DBs; per‑tenant S3 prefixes; per‑tenant policies.
|
||||
* **AuthN/Z**: Authority‑issued OpToks; RBAC roles (`vex.read`, `vex.admin`, `vex.export`).
|
||||
* **No secrets in logs**; deterministic logging contexts include providerId, docDigest, claim keys.
|
||||
|
||||
---
|
||||
|
||||
## 11) Performance & scale
|
||||
|
||||
* **Targets:**
|
||||
|
||||
* Normalize 10k VEX claims/minute/core.
|
||||
* Consensus compute ≤ 50 ms for 1k unique `(vuln, product)` pairs in hot cache.
|
||||
* Export (consensus) 1M rows in ≤ 60 s on 8 cores with streaming writer.
|
||||
|
||||
* **Scaling:**
|
||||
|
||||
* WebService handles control APIs; **Worker** background services (same image) execute fetch/normalize in parallel with rate‑limits; Mongo writes batched; upserts by natural keys.
|
||||
* Exports stream straight to S3 (MinIO) with rolling buffers.
|
||||
|
||||
* **Caching:**
|
||||
|
||||
* `vex.cache` maps query signatures → export; TTL to avoid stampedes; optimistic reuse unless `force`.
|
||||
|
||||
---
|
||||
|
||||
## 12) Observability
|
||||
|
||||
* **Metrics:**
|
||||
|
||||
* `vex.ingest.docs_total{provider}`
|
||||
* `vex.normalize.claims_total{provider}`
|
||||
* `vex.signature.failures_total{provider,method}`
|
||||
* `vex.consensus.conflicts_total{vulnId}`
|
||||
* `vex.exports.bytes{format}` / `vex.exports.latency_seconds`
|
||||
* **Tracing:** spans for fetch, verify, parse, map, consensus, export.
|
||||
* **Dashboards:** provider staleness, top conflicting vulns/components, signature posture, export cache hit‑rate.
|
||||
|
||||
---
|
||||
|
||||
## 13) Testing matrix
|
||||
|
||||
* **Connectors:** golden raw docs → deterministic claims (fixtures per provider/format).
|
||||
* **Signature policies:** valid/invalid PGP/cosign/x509 samples; ensure rejects are recorded but not accepted.
|
||||
* **Normalization edge cases:** platform‑only claims, free‑text justifications, non‑purl products.
|
||||
* **Consensus:** conflict scenarios across tiers; check tie‑breakers; justification gates.
|
||||
* **Performance:** 1M‑row export timing; memory ceilings; stream correctness.
|
||||
* **Determinism:** same inputs + policy → identical `consensusDigest` and export bytes.
|
||||
* **API contract tests:** pagination, filters, RBAC, rate limits.
|
||||
|
||||
---
|
||||
|
||||
## 14) Integration points
|
||||
|
||||
* **Backend Policy Engine** (in Scanner.WebService): calls `POST /excititor/resolve` (scope `vex.read`) with batched `(purl, vulnId)` pairs to fetch `rollupStatus + sources`.
|
||||
* **Feedser**: provides alias graph (CVE↔vendor IDs) and may supply VEX‑adjacent metadata (e.g., KEV flag) for policy escalation.
|
||||
* **UI**: VEX explorer screens use `/claims/search` and `/consensus/search`; show conflicts & provenance.
|
||||
* **CLI**: `stellaops vex export --consensus --since 7d --out vex.json` for audits.
|
||||
|
||||
---
|
||||
|
||||
## 15) Failure modes & fallback
|
||||
|
||||
* **Provider unreachable:** stale thresholds trigger warnings; policy can down‑weight stale providers automatically (freshness factor).
|
||||
* **Signature outage:** continue to ingest but mark `signatureState.verified=false`; consensus will likely exclude or down‑weight per policy.
|
||||
* **Schema drift:** unknown fields are preserved as `evidence`; normalization rejects only on **invalid identity** or **status**.
|
||||
|
||||
---
|
||||
|
||||
## 16) Rollout plan (incremental)
|
||||
|
||||
1. **MVP**: OpenVEX + CSAF connectors for 3 major providers (e.g., Red Hat/SUSE/Ubuntu), normalization + consensus + `/excititor/resolve`.
|
||||
2. **Signature policies**: PGP for distros; cosign for OCI.
|
||||
3. **Exports + optional attestation**.
|
||||
4. **CycloneDX VEX** connectors; platform claim expansion tables; UI explorer.
|
||||
5. **Scale hardening**: export indexes; conflict analytics.
|
||||
|
||||
---
|
||||
|
||||
## 17) Appendix — canonical JSON (stable ordering)
|
||||
|
||||
All exports and consensus entries are serialized via `VexCanonicalJsonSerializer`:
|
||||
|
||||
* UTF‑8 without BOM;
|
||||
* keys sorted (ASCII);
|
||||
* arrays sorted by `(providerId, vulnId, productKey, lastObserved)` unless semantic order mandated;
|
||||
* timestamps in `YYYY‑MM‑DDThh:mm:ssZ`;
|
||||
* no insignificant whitespace.
|
||||
|
||||
# component_architecture_vexer.md — **Stella Ops Vexer** (2025Q4)
|
||||
|
||||
> **Scope.** This document specifies the **Vexer** service: its purpose, trust model, data structures, APIs, plug‑in contracts, storage schema, normalization/consensus algorithms, performance budgets, testing matrix, and how it integrates with Scanner, Policy, Feedser, and the attestation chain. It is implementation‑ready.
|
||||
|
||||
---
|
||||
|
||||
## 0) Mission & role in the platform
|
||||
|
||||
**Mission.** Convert heterogeneous **VEX** statements (OpenVEX, CSAF VEX, CycloneDX VEX; vendor/distro/platform sources) into **canonical, queryable claims**; compute **deterministic consensus** per *(vuln, product)*; preserve **conflicts with provenance**; publish **stable, attestable exports** that the backend uses to suppress non‑exploitable findings, prioritize remaining risk, and explain decisions.
|
||||
|
||||
**Boundaries.**
|
||||
|
||||
* Vexer **does not** decide PASS/FAIL. It supplies **evidence** (statuses + justifications + provenance weights).
|
||||
* Vexer preserves **conflicting claims** unchanged; consensus encodes how we would pick, but the raw set is always exportable.
|
||||
* VEX consumption is **backend‑only**: Scanner never applies VEX. The backend’s **Policy Engine** asks Vexer for status evidence and then decides what to show.
|
||||
|
||||
---
|
||||
|
||||
## 1) Inputs, outputs & canonical domain
|
||||
|
||||
### 1.1 Accepted input formats (ingest)
|
||||
|
||||
* **OpenVEX** JSON documents (attested or raw).
|
||||
* **CSAF VEX** 2.x (vendor PSIRTs and distros commonly publish CSAF).
|
||||
* **CycloneDX VEX** 1.4+ (standalone VEX or embedded VEX blocks).
|
||||
* **OCI‑attached attestations** (VEX statements shipped as OCI referrers) — optional connectors.
|
||||
|
||||
All connectors register **source metadata**: provider identity, trust tier, signature expectations (PGP/cosign/PKI), fetch windows, rate limits, and time anchors.
|
||||
|
||||
### 1.2 Canonical model (normalized)
|
||||
|
||||
Every incoming statement becomes a set of **VexClaim** records:
|
||||
|
||||
```
|
||||
VexClaim
|
||||
- providerId // 'redhat', 'suse', 'ubuntu', 'github', 'vendorX'
|
||||
- vulnId // 'CVE-2025-12345', 'GHSA-xxxx', canonicalized
|
||||
- productKey // canonical product identity (see §2.2)
|
||||
- status // affected | not_affected | fixed | under_investigation
|
||||
- justification? // for 'not_affected'/'affected' where provided
|
||||
- introducedVersion? // semantics per provider (range or exact)
|
||||
- fixedVersion? // where provided (range or exact)
|
||||
- lastObserved // timestamp from source or fetch time
|
||||
- provenance // doc digest, signature status, fetch URI, line/offset anchors
|
||||
- evidence[] // raw source snippets for explainability
|
||||
- supersedes? // optional cross-doc chain (docDigest → docDigest)
|
||||
```
|
||||
|
||||
### 1.3 Exports (consumption)
|
||||
|
||||
* **VexConsensus** per `(vulnId, productKey)` with:
|
||||
|
||||
* `rollupStatus` (after policy weights/justification gates),
|
||||
* `sources[]` (winning + losing claims with weights & reasons),
|
||||
* `policyRevisionId` (identifier of the Vexer policy used),
|
||||
* `consensusDigest` (stable SHA‑256 over canonical JSON).
|
||||
* **Raw claims** export for auditing (unchanged, with provenance).
|
||||
* **Provider snapshots** (per source, last N days) for operator debugging.
|
||||
* **Index** optimized for backend joins: `(productKey, vulnId) → (status, confidence, sourceSet)`.
|
||||
|
||||
All exports are **deterministic**, and (optionally) **attested** via DSSE and logged to Rekor v2.
|
||||
|
||||
---
|
||||
|
||||
## 2) Identity model — products & joins
|
||||
|
||||
### 2.1 Vuln identity
|
||||
|
||||
* Accepts **CVE**, **GHSA**, vendor IDs (MSRC, RHSA…), distro IDs (DSA/USN/RHSA…) — normalized to `vulnId` with alias sets.
|
||||
* **Alias graph** maintained (from Feedser) to map vendor/distro IDs → CVE (primary) and to **GHSA** where applicable.
|
||||
|
||||
### 2.2 Product identity (`productKey`)
|
||||
|
||||
* **Primary:** `purl` (Package URL).
|
||||
* **Secondary links:** `cpe`, **OS package NVRA/EVR**, NuGet/Maven/Golang identity, and **OS package name** when purl unavailable.
|
||||
* **Fallback:** `oci:<registry>/<repo>@<digest>` for image‑level VEX.
|
||||
* **Special cases:** kernel modules, firmware, platforms → provider‑specific mapping helpers (connector captures provider’s product taxonomy → canonical `productKey`).
|
||||
|
||||
> Vexer does not invent identities. If a provider cannot be mapped to purl/CPE/NVRA deterministically, we keep the native **product string** and mark the claim as **non‑joinable**; the backend will ignore it unless a policy explicitly whitelists that provider mapping.
|
||||
|
||||
---
|
||||
|
||||
## 3) Storage schema (MongoDB)
|
||||
|
||||
Database: `vexer`
|
||||
|
||||
### 3.1 Collections
|
||||
|
||||
**`vex.providers`**
|
||||
|
||||
```
|
||||
_id: providerId
|
||||
name, homepage, contact
|
||||
trustTier: enum {vendor, distro, platform, hub, attestation}
|
||||
signaturePolicy: { type: pgp|cosign|x509|none, keys[], certs[], cosignKeylessRoots[] }
|
||||
fetch: { baseUrl, kind: http|oci|file, rateLimit, etagSupport, windowDays }
|
||||
enabled: bool
|
||||
createdAt, modifiedAt
|
||||
```
|
||||
|
||||
**`vex.raw`** (immutable raw documents)
|
||||
|
||||
```
|
||||
_id: sha256(doc bytes)
|
||||
providerId
|
||||
uri
|
||||
ingestedAt
|
||||
contentType
|
||||
sig: { verified: bool, method: pgp|cosign|x509|none, keyId|certSubject, bundle? }
|
||||
payload: GridFS pointer (if large)
|
||||
disposition: kept|replaced|superseded
|
||||
correlation: { replaces?: sha256, replacedBy?: sha256 }
|
||||
```
|
||||
|
||||
**`vex.claims`** (normalized rows; dedupe on providerId+vulnId+productKey+docDigest)
|
||||
|
||||
```
|
||||
_id
|
||||
providerId
|
||||
vulnId
|
||||
productKey
|
||||
status
|
||||
justification?
|
||||
introducedVersion?
|
||||
fixedVersion?
|
||||
lastObserved
|
||||
docDigest
|
||||
provenance { uri, line?, pointer?, signatureState }
|
||||
evidence[] { key, value, locator }
|
||||
indices:
|
||||
- {vulnId:1, productKey:1}
|
||||
- {providerId:1, lastObserved:-1}
|
||||
- {status:1}
|
||||
- text index (optional) on evidence.value for debugging
|
||||
```
|
||||
|
||||
**`vex.consensus`** (rollups)
|
||||
|
||||
```
|
||||
_id: sha256(canonical(vulnId, productKey, policyRevision))
|
||||
vulnId
|
||||
productKey
|
||||
rollupStatus
|
||||
sources[]: [
|
||||
{ providerId, status, justification?, weight, lastObserved, accepted:bool, reason }
|
||||
]
|
||||
policyRevisionId
|
||||
evaluatedAt
|
||||
consensusDigest // same as _id
|
||||
indices:
|
||||
- {vulnId:1, productKey:1}
|
||||
- {policyRevisionId:1, evaluatedAt:-1}
|
||||
```
|
||||
|
||||
**`vex.exports`** (manifest of emitted artifacts)
|
||||
|
||||
```
|
||||
_id
|
||||
querySignature
|
||||
format: raw|consensus|index
|
||||
artifactSha256
|
||||
rekor { uuid, index, url }?
|
||||
createdAt
|
||||
policyRevisionId
|
||||
cacheable: bool
|
||||
```
|
||||
|
||||
**`vex.cache`**
|
||||
|
||||
```
|
||||
querySignature -> exportId (for fast reuse)
|
||||
ttl, hits
|
||||
```
|
||||
|
||||
**`vex.migrations`**
|
||||
|
||||
* ordered migrations applied at bootstrap to ensure indexes.
|
||||
|
||||
### 3.2 Indexing strategy
|
||||
|
||||
* Hot path queries use exact `(vulnId, productKey)` and time‑bounded windows; compound indexes cover both.
|
||||
* Providers list view by `lastObserved` for monitoring staleness.
|
||||
* `vex.consensus` keyed by `(vulnId, productKey, policyRevision)` for deterministic reuse.
|
||||
|
||||
---
|
||||
|
||||
## 4) Ingestion pipeline
|
||||
|
||||
### 4.1 Connector contract
|
||||
|
||||
```csharp
|
||||
public interface IVexConnector
|
||||
{
|
||||
string ProviderId { get; }
|
||||
Task FetchAsync(VexConnectorContext ctx, CancellationToken ct); // raw docs
|
||||
Task NormalizeAsync(VexConnectorContext ctx, CancellationToken ct); // raw -> VexClaim[]
|
||||
}
|
||||
```
|
||||
|
||||
* **Fetch** must implement: window scheduling, conditional GET (ETag/If‑Modified‑Since), rate limiting, retry/backoff.
|
||||
* **Normalize** parses the format, validates schema, maps product identities deterministically, emits `VexClaim` records with **provenance**.
|
||||
|
||||
### 4.2 Signature verification (per provider)
|
||||
|
||||
* **cosign (keyless or keyful)** for OCI referrers or HTTP‑served JSON with Sigstore bundles.
|
||||
* **PGP** (provider keyrings) for distro/vendor feeds that sign docs.
|
||||
* **x509** (mutual TLS / provider‑pinned certs) where applicable.
|
||||
* Signature state is stored on **vex.raw.sig** and copied into **provenance.signatureState** on claims.
|
||||
|
||||
> Claims from sources failing signature policy are marked `"signatureState.verified=false"` and **policy** can down‑weight or ignore them.
|
||||
|
||||
### 4.3 Time discipline
|
||||
|
||||
* For each doc, prefer **provider’s document timestamp**; if absent, use fetch time.
|
||||
* Claims carry `lastObserved` which drives **tie‑breaking** within equal weight tiers.
|
||||
|
||||
---
|
||||
|
||||
## 5) Normalization: product & status semantics
|
||||
|
||||
### 5.1 Product mapping
|
||||
|
||||
* **purl** first; **cpe** second; OS package NVRA/EVR mapping helpers (distro connectors) produce purls via canonical tables (e.g., rpm→purl:rpm, deb→purl:deb).
|
||||
* Where a provider publishes **platform‑level** VEX (e.g., “RHEL 9 not affected”), connectors expand to known product inventory rules (e.g., map to sets of packages/components shipped in the platform). Expansion tables are versioned and kept per provider; every expansion emits **evidence** indicating the rule applied.
|
||||
* If expansion would be speculative, the claim remains **platform‑scoped** with `productKey="platform:redhat:rhel:9"` and is flagged **non‑joinable**; backend can decide to use platform VEX only when Scanner proves the platform runtime.
|
||||
|
||||
### 5.2 Status + justification mapping
|
||||
|
||||
* Canonical **status**: `affected | not_affected | fixed | under_investigation`.
|
||||
* **Justifications** normalized to a controlled vocabulary (CISA‑aligned), e.g.:
|
||||
|
||||
* `component_not_present`
|
||||
* `vulnerable_code_not_in_execute_path`
|
||||
* `vulnerable_configuration_unused`
|
||||
* `inline_mitigation_applied`
|
||||
* `fix_available` (with `fixedVersion`)
|
||||
* `under_investigation`
|
||||
* Providers with free‑text justifications are mapped by deterministic tables; raw text preserved as `evidence`.
|
||||
|
||||
---
|
||||
|
||||
## 6) Consensus algorithm
|
||||
|
||||
**Goal:** produce a **stable**, explainable `rollupStatus` per `(vulnId, productKey)` given possibly conflicting claims.
|
||||
|
||||
### 6.1 Inputs
|
||||
|
||||
* Set **S** of `VexClaim` for the key.
|
||||
* **Vexer policy snapshot**:
|
||||
|
||||
* **weights** per provider tier and per provider overrides.
|
||||
* **justification gates** (e.g., require justification for `not_affected` to be acceptable).
|
||||
* **minEvidence** rules (e.g., `not_affected` must come from ≥1 vendor or 2 distros).
|
||||
* **signature requirements** (e.g., require verified signature for ‘fixed’ to be considered).
|
||||
|
||||
### 6.2 Steps
|
||||
|
||||
1. **Filter invalid** claims by signature policy & justification gates → set `S'`.
|
||||
2. **Score** each claim:
|
||||
`score = weight(provider) * freshnessFactor(lastObserved)` where freshnessFactor ∈ [0.8, 1.0] for staleness decay (configurable; small effect).
|
||||
3. **Aggregate** scores per status: `W(status) = Σ score(claims with that status)`.
|
||||
4. **Pick** `rollupStatus = argmax_status W(status)`.
|
||||
5. **Tie‑breakers** (in order):
|
||||
|
||||
* Higher **max single** provider score wins (vendor > distro > platform > hub).
|
||||
* More **recent** lastObserved wins.
|
||||
* Deterministic lexicographic order of status (`fixed` > `not_affected` > `under_investigation` > `affected`) as final tiebreaker.
|
||||
6. **Explain**: mark accepted sources (`accepted=true; reason="weight"`/`"freshness"`), mark rejected sources with explicit `reason` (`"insufficient_justification"`, `"signature_unverified"`, `"lower_weight"`).
|
||||
|
||||
> The algorithm is **pure** given S and policy snapshot; result is reproducible and hashed into `consensusDigest`.
|
||||
|
||||
---
|
||||
|
||||
## 7) Query & export APIs
|
||||
|
||||
All endpoints are versioned under `/api/v1/vex`.
|
||||
|
||||
### 7.1 Query (online)
|
||||
|
||||
```
|
||||
POST /claims/search
|
||||
body: { vulnIds?: string[], productKeys?: string[], providers?: string[], since?: timestamp, limit?: int, pageToken?: string }
|
||||
→ { claims[], nextPageToken? }
|
||||
|
||||
POST /consensus/search
|
||||
body: { vulnIds?: string[], productKeys?: string[], policyRevisionId?: string, since?: timestamp, limit?: int, pageToken?: string }
|
||||
→ { entries[], nextPageToken? }
|
||||
|
||||
POST /excititor/resolve (scope: vex.read)
|
||||
body: { productKeys?: string[], purls?: string[], vulnerabilityIds: string[], policyRevisionId?: string }
|
||||
→ { policy, resolvedAt, results: [ { vulnerabilityId, productKey, status, sources[], conflicts[], decisions[], signals?, summary?, envelope: { artifact, contentSignature?, attestation?, attestationEnvelope?, attestationSignature? } } ] }
|
||||
```
|
||||
|
||||
### 7.2 Exports (cacheable snapshots)
|
||||
|
||||
```
|
||||
POST /exports
|
||||
body: { signature: { vulnFilter?, productFilter?, providers?, since? }, format: raw|consensus|index, policyRevisionId?: string, force?: bool }
|
||||
→ { exportId, artifactSha256, rekor? }
|
||||
|
||||
GET /exports/{exportId} → bytes (application/json or binary index)
|
||||
GET /exports/{exportId}/meta → { signature, policyRevisionId, createdAt, artifactSha256, rekor? }
|
||||
```
|
||||
|
||||
### 7.3 Provider operations
|
||||
|
||||
```
|
||||
GET /providers → provider list & signature policy
|
||||
POST /providers/{id}/refresh → trigger fetch/normalize window
|
||||
GET /providers/{id}/status → last fetch, doc counts, signature stats
|
||||
```
|
||||
|
||||
**Auth:** service‑to‑service via Authority tokens; operator operations via UI/CLI with RBAC.
|
||||
|
||||
---
|
||||
|
||||
## 8) Attestation integration
|
||||
|
||||
* Exports can be **DSSE‑signed** via **Signer** and logged to **Rekor v2** via **Attestor** (optional but recommended for regulated pipelines).
|
||||
* `vex.exports.rekor` stores `{uuid, index, url}` when present.
|
||||
* **Predicate type**: `https://stella-ops.org/attestations/vex-export/1` with fields:
|
||||
|
||||
* `querySignature`, `policyRevisionId`, `artifactSha256`, `createdAt`.
|
||||
|
||||
---
|
||||
|
||||
## 9) Configuration (YAML)
|
||||
|
||||
```yaml
|
||||
vexer:
|
||||
mongo: { uri: "mongodb://mongo/vexer" }
|
||||
s3:
|
||||
endpoint: http://minio:9000
|
||||
bucket: stellaops
|
||||
policy:
|
||||
weights:
|
||||
vendor: 1.0
|
||||
distro: 0.9
|
||||
platform: 0.7
|
||||
hub: 0.5
|
||||
attestation: 0.6
|
||||
providerOverrides:
|
||||
redhat: 1.0
|
||||
suse: 0.95
|
||||
requireJustificationForNotAffected: true
|
||||
signatureRequiredForFixed: true
|
||||
minEvidence:
|
||||
not_affected:
|
||||
vendorOrTwoDistros: true
|
||||
connectors:
|
||||
- providerId: redhat
|
||||
kind: csaf
|
||||
baseUrl: https://access.redhat.com/security/data/csaf/v2/
|
||||
signaturePolicy: { type: pgp, keys: [ "…redhat-pgp-key…" ] }
|
||||
windowDays: 7
|
||||
- providerId: suse
|
||||
kind: csaf
|
||||
baseUrl: https://ftp.suse.com/pub/projects/security/csaf/
|
||||
signaturePolicy: { type: pgp, keys: [ "…suse-pgp-key…" ] }
|
||||
- providerId: ubuntu
|
||||
kind: openvex
|
||||
baseUrl: https://…/vex/
|
||||
signaturePolicy: { type: none }
|
||||
- providerId: vendorX
|
||||
kind: cyclonedx-vex
|
||||
ociRef: ghcr.io/vendorx/vex@sha256:…
|
||||
signaturePolicy: { type: cosign, cosignKeylessRoots: [ "sigstore-root" ] }
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 10) Security model
|
||||
|
||||
* **Input signature verification** enforced per provider policy (PGP, cosign, x509).
|
||||
* **Connector allowlists**: outbound fetch constrained to configured domains.
|
||||
* **Tenant isolation**: per‑tenant DB prefixes or separate DBs; per‑tenant S3 prefixes; per‑tenant policies.
|
||||
* **AuthN/Z**: Authority‑issued OpToks; RBAC roles (`vex.read`, `vex.admin`, `vex.export`).
|
||||
* **No secrets in logs**; deterministic logging contexts include providerId, docDigest, claim keys.
|
||||
|
||||
---
|
||||
|
||||
## 11) Performance & scale
|
||||
|
||||
* **Targets:**
|
||||
|
||||
* Normalize 10k VEX claims/minute/core.
|
||||
* Consensus compute ≤ 50 ms for 1k unique `(vuln, product)` pairs in hot cache.
|
||||
* Export (consensus) 1M rows in ≤ 60 s on 8 cores with streaming writer.
|
||||
|
||||
* **Scaling:**
|
||||
|
||||
* WebService handles control APIs; **Worker** background services (same image) execute fetch/normalize in parallel with rate‑limits; Mongo writes batched; upserts by natural keys.
|
||||
* Exports stream straight to S3 (MinIO) with rolling buffers.
|
||||
|
||||
* **Caching:**
|
||||
|
||||
* `vex.cache` maps query signatures → export; TTL to avoid stampedes; optimistic reuse unless `force`.
|
||||
|
||||
---
|
||||
|
||||
## 12) Observability
|
||||
|
||||
* **Metrics:**
|
||||
|
||||
* `vex.ingest.docs_total{provider}`
|
||||
* `vex.normalize.claims_total{provider}`
|
||||
* `vex.signature.failures_total{provider,method}`
|
||||
* `vex.consensus.conflicts_total{vulnId}`
|
||||
* `vex.exports.bytes{format}` / `vex.exports.latency_seconds`
|
||||
* **Tracing:** spans for fetch, verify, parse, map, consensus, export.
|
||||
* **Dashboards:** provider staleness, top conflicting vulns/components, signature posture, export cache hit‑rate.
|
||||
|
||||
---
|
||||
|
||||
## 13) Testing matrix
|
||||
|
||||
* **Connectors:** golden raw docs → deterministic claims (fixtures per provider/format).
|
||||
* **Signature policies:** valid/invalid PGP/cosign/x509 samples; ensure rejects are recorded but not accepted.
|
||||
* **Normalization edge cases:** platform‑only claims, free‑text justifications, non‑purl products.
|
||||
* **Consensus:** conflict scenarios across tiers; check tie‑breakers; justification gates.
|
||||
* **Performance:** 1M‑row export timing; memory ceilings; stream correctness.
|
||||
* **Determinism:** same inputs + policy → identical `consensusDigest` and export bytes.
|
||||
* **API contract tests:** pagination, filters, RBAC, rate limits.
|
||||
|
||||
---
|
||||
|
||||
## 14) Integration points
|
||||
|
||||
* **Backend Policy Engine** (in Scanner.WebService): calls `POST /excititor/resolve` (scope `vex.read`) with batched `(purl, vulnId)` pairs to fetch `rollupStatus + sources`.
|
||||
* **Feedser**: provides alias graph (CVE↔vendor IDs) and may supply VEX‑adjacent metadata (e.g., KEV flag) for policy escalation.
|
||||
* **UI**: VEX explorer screens use `/claims/search` and `/consensus/search`; show conflicts & provenance.
|
||||
* **CLI**: `stellaops vex export --consensus --since 7d --out vex.json` for audits.
|
||||
|
||||
---
|
||||
|
||||
## 15) Failure modes & fallback
|
||||
|
||||
* **Provider unreachable:** stale thresholds trigger warnings; policy can down‑weight stale providers automatically (freshness factor).
|
||||
* **Signature outage:** continue to ingest but mark `signatureState.verified=false`; consensus will likely exclude or down‑weight per policy.
|
||||
* **Schema drift:** unknown fields are preserved as `evidence`; normalization rejects only on **invalid identity** or **status**.
|
||||
|
||||
---
|
||||
|
||||
## 16) Rollout plan (incremental)
|
||||
|
||||
1. **MVP**: OpenVEX + CSAF connectors for 3 major providers (e.g., Red Hat/SUSE/Ubuntu), normalization + consensus + `/excititor/resolve`.
|
||||
2. **Signature policies**: PGP for distros; cosign for OCI.
|
||||
3. **Exports + optional attestation**.
|
||||
4. **CycloneDX VEX** connectors; platform claim expansion tables; UI explorer.
|
||||
5. **Scale hardening**: export indexes; conflict analytics.
|
||||
|
||||
---
|
||||
|
||||
## 17) Appendix — canonical JSON (stable ordering)
|
||||
|
||||
All exports and consensus entries are serialized via `VexCanonicalJsonSerializer`:
|
||||
|
||||
* UTF‑8 without BOM;
|
||||
* keys sorted (ASCII);
|
||||
* arrays sorted by `(providerId, vulnId, productKey, lastObserved)` unless semantic order mandated;
|
||||
* timestamps in `YYYY‑MM‑DDThh:mm:ssZ`;
|
||||
* no insignificant whitespace.
|
||||
|
||||
|
||||
@@ -82,7 +82,7 @@ Everything here is open‑source and versioned — when you check out a git ta
|
||||
- **70a – [Policy Gateway](policy/gateway.md)**
|
||||
- **71 – [Policy Examples](examples/policies/README.md)**
|
||||
- **72 – [Policy FAQ](faq/policy-faq.md)**
|
||||
- **73 – [Policy Run DTOs](../src/StellaOps.Scheduler.Models/docs/SCHED-MODELS-20-001-POLICY-RUNS.md)**
|
||||
- **73 – [Policy Run DTOs](../src/Scheduler/__Libraries/StellaOps.Scheduler.Models/docs/SCHED-MODELS-20-001-POLICY-RUNS.md)**
|
||||
- **30 – [Fixture Maintenance](dev/fixtures.md)**
|
||||
- **74 – [Export Center Overview](export-center/overview.md)**
|
||||
- **75 – [Export Center Architecture](export-center/architecture.md)**
|
||||
@@ -147,10 +147,10 @@ Everything here is open‑source and versioned — when you check out a git ta
|
||||
|
||||
> Imposed rule: Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.
|
||||
|
||||
- **Aggregation-Only Contract (AOC).** Ingestion services aggregate and link facts only—derived precedence, severity, and safe-fix hints live in Policy overlays and dedicated explorers. Review [`../AGENTS.md`](../AGENTS.md) and the AOC guardrails in [`aoc/aoc-guardrails.md`](aoc/aoc-guardrails.md).
|
||||
- **Aggregation-Only Contract (AOC).** Ingestion services aggregate and link facts only—derived precedence, severity, and safe-fix hints live in Policy overlays and dedicated explorers. Review [`implplan/AGENTS.md`](implplan/AGENTS.md) and the AOC guardrails in [`aoc/aoc-guardrails.md`](aoc/aoc-guardrails.md).
|
||||
- **Cartographer owns graphs.** SBOM Service emits projections/events; Cartographer (`CARTO-GRAPH-21-00x`) builds graph storage, overlays, and tiles. See `ARCHITECTURE_CONCELIER.md` (Cartographer handshake section) for handoff boundaries.
|
||||
- **Notifier replaces legacy Notify.** Sprint‑15 `StellaOps.Notify.*` tasks are frozen; use the Notifications Studio/Notifier backlogs (`NOTIFY-SVC-38..40`, `WEB-NOTIFY-3x-00x`, `CLI-NOTIFY-3x-00x`).
|
||||
- **Dedicated services for Vuln & Policy.** Vuln Explorer work flows through `src/StellaOps.VulnExplorer.Api`/Console/CLI (Sprint 29); gateway routes proxy only. Policy Engine remains the sole source for precedence/suppression overlays.
|
||||
- **Dedicated services for Vuln & Policy.** Vuln Explorer work flows through `src/VulnExplorer/StellaOps.VulnExplorer.Api`/Console/CLI (Sprint 29); gateway routes proxy only. Policy Engine remains the sole source for precedence/suppression overlays.
|
||||
- **Cleanup log.** The backlog consolidation summary lives in [`backlog/2025-10-cleanup.md`](backlog/2025-10-cleanup.md).
|
||||
|
||||
© 2025 Stella Ops contributors – licensed AGPL‑3.0‑or‑later
|
||||
|
||||
762
docs/TASKS.md
762
docs/TASKS.md
@@ -1,381 +1,381 @@
|
||||
# Docs Guild Task Board (UTC 2025-10-10)
|
||||
|
||||
| ID | Status | Owner(s) | Depends on | Description | Exit Criteria |
|
||||
|----|--------|----------|------------|-------------|---------------|
|
||||
| DOC7.README-INDEX | DONE (2025-10-17) | Docs Guild | — | Refresh index docs (docs/README.md + root README) after architecture dossier split and Offline Kit overhaul. | ✅ ToC reflects new component architecture docs; ✅ root README highlights updated doc set; ✅ Offline Kit guide linked correctly. |
|
||||
| DOC4.AUTH-PDG | DONE (2025-10-19) | Docs Guild, Plugin Team | PLG6.DOC | Copy-edit `docs/dev/31_AUTHORITY_PLUGIN_DEVELOPER_GUIDE.md`, export lifecycle diagram, add LDAP RFC cross-link. | ✅ PR merged with polish; ✅ Diagram committed; ✅ Slack handoff posted. |
|
||||
| DOC1.AUTH | DONE (2025-10-12) | Docs Guild, Authority Core | CORE5B.DOC | Draft `docs/11_AUTHORITY.md` covering architecture, configuration, bootstrap flows. | ✅ Architecture + config sections approved by Core; ✅ Samples reference latest options; ✅ Offline note added. |
|
||||
| DOC3.Concelier-Authority | DONE (2025-10-12) | Docs Guild, DevEx | FSR4 | Polish operator/runbook sections (DOC3/DOC5) to document Concelier authority rollout, bypass logging, and enforcement checklist. | ✅ DOC3/DOC5 updated with audit runbook references; ✅ enforcement deadline highlighted; ✅ Docs guild sign-off. |
|
||||
| DOC5.Concelier-Runbook | DONE (2025-10-12) | Docs Guild | DOC3.Concelier-Authority | Produce dedicated Concelier authority audit runbook covering log fields, monitoring recommendations, and troubleshooting steps. | ✅ Runbook published; ✅ linked from DOC3/DOC5; ✅ alerting guidance included. |
|
||||
| FEEDDOCS-DOCS-05-001 | DONE (2025-10-11) | Docs Guild | FEEDMERGE-ENGINE-04-001, FEEDMERGE-ENGINE-04-002 | Publish Concelier conflict resolution runbook covering precedence workflow, merge-event auditing, and Sprint 3 metrics. | ✅ `docs/ops/concelier-conflict-resolution.md` committed; ✅ metrics/log tables align with latest merge code; ✅ Ops alert guidance handed to Concelier team. |
|
||||
| FEEDDOCS-DOCS-05-002 | DONE (2025-10-16) | Docs Guild, Concelier Ops | FEEDDOCS-DOCS-05-001 | Ops sign-off captured: conflict runbook circulated, alert thresholds tuned, and rollout decisions documented in change log. | ✅ Ops review recorded; ✅ alert thresholds finalised using `docs/ops/concelier-authority-audit-runbook.md`; ✅ change-log entry linked from runbook once GHSA/NVD/OSV regression fixtures land. |
|
||||
| DOCS-ADR-09-001 | DONE (2025-10-19) | Docs Guild, DevEx | — | Establish ADR process (`docs/adr/0000-template.md`) and document usage guidelines. | Template published; README snippet linking ADR process; announcement posted (`docs/updates/2025-10-18-docs-guild.md`). |
|
||||
| DOCS-EVENTS-09-002 | DONE (2025-10-19) | Docs Guild, Platform Events | SCANNER-EVENTS-15-201 | Publish event schema catalog (`docs/events/`) for `scanner.report.ready@1`, `scheduler.rescan.delta@1`, `attestor.logged@1`. | Schemas validated (Ajv CI hooked); docs/events/README summarises usage; Platform Events notified via `docs/updates/2025-10-18-docs-guild.md`. |
|
||||
| DOCS-EVENTS-09-003 | DONE (2025-10-19) | Docs Guild | DOCS-EVENTS-09-002 | Add human-readable envelope field references and canonical payload samples for published events, including offline validation workflow. | Tables explain common headers/payload segments; versioned sample payloads committed; README links to validation instructions and samples. |
|
||||
| DOCS-EVENTS-09-004 | DONE (2025-10-19) | Docs Guild, Scanner WebService | SCANNER-EVENTS-15-201 | Refresh scanner event docs to mirror DSSE-backed report fields, document `scanner.scan.completed`, and capture canonical sample validation. | Schemas updated for new payload shape; README references DSSE reuse and validation test; samples align with emitted events. |
|
||||
| PLATFORM-EVENTS-09-401 | DONE (2025-10-21) | Platform Events Guild | DOCS-EVENTS-09-003 | Embed canonical event samples into contract/integration tests and ensure CI validates payloads against published schemas. | Notify models tests now run schema validation against `docs/events/*.json`, event schemas allow optional `attributes`, and docs capture the new validation workflow. |
|
||||
| RUNTIME-GUILD-09-402 | DONE (2025-10-19) | Runtime Guild | SCANNER-POLICY-09-107 | Confirm Scanner WebService surfaces `quietedFindingCount` and progress hints to runtime consumers; document readiness checklist. | Runtime verification run captures enriched payload; checklist/doc updates merged; stakeholders acknowledge availability. |
|
||||
| DOCS-CONCELIER-07-201 | DONE (2025-10-22) | Docs Guild, Concelier WebService | FEEDWEB-DOCS-01-001 | Final editorial review and publish pass for Concelier authority toggle documentation (Quickstart + operator guide). | Review feedback resolved, publish PR merged, release notes updated with documentation pointer. |
|
||||
| DOCS-RUNTIME-17-004 | DONE (2025-10-26) | Docs Guild, Runtime Guild | SCANNER-EMIT-17-701, ZASTAVA-OBS-17-005, DEVOPS-REL-17-002 | Document build-id workflows: SBOM exposure, runtime event payloads (`process.buildId`), Scanner `/policy/runtime` response (`buildIds` list), debug-store layout, and operator guidance for symbol retrieval. | Architecture + operator docs updated with build-id sections (Observer, Scanner, CLI), examples show `readelf` output + debuginfod usage, references linked from Offline Kit/Release guides + CLI help. |
|
||||
| DOCS-OBS-50-001 | BLOCKED (2025-10-26) | Docs Guild, Observability Guild | TELEMETRY-OBS-50-001 | Publish `/docs/observability/overview.md` introducing scope, imposed rule banner, architecture diagram, and tenant guarantees. | Doc merged with imposed rule banner; diagram committed; cross-links to telemetry stack + evidence locker docs. |
|
||||
> Blocked: waiting on telemetry core deliverable (TELEMETRY-OBS-50-001) to finalise architecture details and diagrams.
|
||||
| DOCS-OBS-50-002 | TODO | Docs Guild, Security Guild | TELEMETRY-OBS-50-002 | Author `/docs/observability/telemetry-standards.md` detailing common fields, scrubbing policy, sampling defaults, and redaction override procedure. | Doc merged; imposed rule banner present; examples validated with telemetry fixtures; security review sign-off captured. |
|
||||
| DOCS-OBS-50-003 | TODO | Docs Guild, Observability Guild | TELEMETRY-OBS-50-001 | Create `/docs/observability/logging.md` covering structured log schema, dos/don'ts, tenant isolation, and copyable examples. | Doc merged with banner; sample logs redacted; lint passes; linked from coding standards. |
|
||||
| DOCS-OBS-50-004 | TODO | Docs Guild, Observability Guild | TELEMETRY-OBS-50-002 | Draft `/docs/observability/tracing.md` explaining context propagation, async linking, CLI header usage, and sampling strategies. | Doc merged; imposed rule banner included; diagrams updated; references to CLI/Console features added. |
|
||||
| DOCS-OBS-51-001 | TODO | Docs Guild, DevOps Guild | WEB-OBS-51-001, DEVOPS-OBS-51-001 | Publish `/docs/observability/metrics-and-slos.md` cataloging metrics, SLO targets, burn rate policies, and alert runbooks. | Doc merged with banner; SLO tables verified; alert workflows linked to incident runbook. |
|
||||
| DOCS-SEC-OBS-50-001 | TODO | Docs Guild, Security Guild | TELEMETRY-OBS-51-002 | Update `/docs/security/redaction-and-privacy.md` to cover telemetry privacy controls, tenant opt-in debug, and imposed rule reminder. | Doc merged; redaction matrix updated; banner present; security sign-off recorded. |
|
||||
| DOCS-INSTALL-50-001 | TODO | Docs Guild, DevOps Guild | DEVOPS-OBS-50-003 | Add `/docs/install/telemetry-stack.md` with collector deployment, exporter options, offline kit notes, and imposed rule banner. | Doc merged; install steps verified on air-gapped profile; banner present; screenshots attached. |
|
||||
| DOCS-FORENSICS-53-001 | TODO | Docs Guild, Evidence Locker Guild | EVID-OBS-53-003 | Publish `/docs/forensics/evidence-locker.md` describing bundle formats, WORM options, retention, legal hold, and imposed rule banner. | Doc merged; manifest examples validated; banner present; legal hold steps aligned with API. |
|
||||
| DOCS-FORENSICS-53-002 | TODO | Docs Guild, Provenance Guild | PROV-OBS-54-001 | Release `/docs/forensics/provenance-attestation.md` covering DSSE schema, signing process, verification workflow, and imposed rule banner. | Doc merged; sample statements reference fixtures; banner included; verification steps tested. |
|
||||
| DOCS-FORENSICS-53-003 | TODO | Docs Guild, Timeline Indexer Guild | TIMELINE-OBS-52-003 | Publish `/docs/forensics/timeline.md` with schema, event kinds, filters, query examples, and imposed rule banner. | Doc merged; query examples validated; banner present; linked from Console/CLI docs. |
|
||||
| DOCS-CONSOLE-OBS-52-001 | TODO | Docs Guild, Console Guild | CONSOLE-OBS-51-001 | Document `/docs/console/observability.md` showcasing Observability Hub widgets, trace/log search, imposed rule banner, and accessibility tips. | Doc merged; screenshots updated; banner present; navigation steps verified. |
|
||||
| DOCS-CONSOLE-OBS-52-002 | TODO | Docs Guild, Console Guild | CONSOLE-OBS-52-002, CONSOLE-OBS-53-001 | Publish `/docs/console/forensics.md` covering timeline explorer, evidence viewer, attestation verifier, imposed rule banner, and troubleshooting. | Doc merged; banner included; workflows validated via Playwright capture; troubleshooting section populated. |
|
||||
| DOCS-CLI-OBS-52-001 | TODO | Docs Guild, DevEx/CLI Guild | CLI-OBS-52-001 | Create `/docs/cli/observability.md` detailing `stella obs` commands, examples, exit codes, imposed rule banner, and scripting tips. | Doc merged; examples tested; banner included; CLI parity matrix updated. |
|
||||
| DOCS-CLI-FORENSICS-53-001 | TODO | Docs Guild, DevEx/CLI Guild | CLI-FORENSICS-54-001 | Publish `/docs/cli/forensics.md` for snapshot/verify/attest commands with sample outputs, imposed rule banner, and offline workflows. | Doc merged; sample bundles verified; banner present; offline notes cross-linked. |
|
||||
| DOCS-RUNBOOK-55-001 | TODO | Docs Guild, Ops Guild | DEVOPS-OBS-55-001, WEB-OBS-55-001 | Author `/docs/runbooks/incidents.md` describing incident mode activation, escalation steps, retention impact, verification checklist, and imposed rule banner. | Doc merged; runbook rehearsed; banner included; linked from alerts. |
|
||||
| DOCS-AOC-19-001 | DONE (2025-10-26) | Docs Guild, Concelier Guild | CONCELIER-WEB-AOC-19-001, EXCITITOR-WEB-AOC-19-001 | Author `/docs/ingestion/aggregation-only-contract.md` covering philosophy, invariants, schemas, error codes, migration, observability, and security checklist. | New doc published with compliance checklist; cross-links from existing docs added. |
|
||||
| DOCS-AOC-19-002 | DONE (2025-10-26) | Docs Guild, Architecture Guild | DOCS-AOC-19-001 | Update `/docs/architecture/overview.md` to include AOC boundary, raw stores, and sequence diagram (fetch → guard → raw insert → policy evaluation). | Overview doc updated with diagrams/text; lint passes; stakeholders sign off. |
|
||||
| DOCS-AOC-19-003 | DONE (2025-10-26) | Docs Guild, Policy Guild | POLICY-AOC-19-003 | Refresh `/docs/architecture/policy-engine.md` clarifying ingestion boundary, raw inputs, and policy-only derived data. | Doc highlights raw-only ingestion contract, updated diagrams merge, compliance checklist added. |
|
||||
| DOCS-AOC-19-004 | DONE (2025-10-26) | Docs Guild, UI Guild | UI-AOC-19-001 | Extend `/docs/ui/console.md` with Sources dashboard tiles, violation drill-down workflow, and verification action. | UI doc updated with screenshots/flow descriptions, compliance checklist appended. |
|
||||
> DOCS-AOC-19-004: Architecture overview & policy-engine updates landed 2025-10-26; incorporate the new AOC boundary diagrams and metrics references.
|
||||
| DOCS-AOC-19-005 | DONE (2025-10-26) | Docs Guild, CLI Guild | CLI-AOC-19-003 | Update `/docs/cli/cli-reference.md` with `stella sources ingest --dry-run` and `stella aoc verify` usage, exit codes, and offline notes. | CLI reference + quickstart sections updated; examples validated; compliance checklist added. |
|
||||
> DOCS-AOC-19-005: New ingestion reference + architecture overview published 2025-10-26; ensure CLI docs link to both and surface AOC exit codes mapping.
|
||||
| DOCS-AOC-19-006 | DONE (2025-10-26) | Docs Guild, Observability Guild | CONCELIER-WEB-AOC-19-002, EXCITITOR-WEB-AOC-19-002 | Document new metrics/traces/log keys in `/docs/observability/observability.md`. | Observability doc lists new metrics/traces/log fields; dashboards referenced; compliance checklist appended. |
|
||||
| DOCS-AOC-19-007 | DONE (2025-10-26) | Docs Guild, Authority Core | AUTH-AOC-19-001 | Update `/docs/security/authority-scopes.md` with new ingestion scopes and tenancy enforcement notes. | Doc reflects new scopes, sample policies updated, compliance checklist added. |
|
||||
| DOCS-AOC-19-008 | DONE (2025-10-26) | Docs Guild, DevOps Guild | DEVOPS-AOC-19-002 | Refresh `/docs/deploy/containers.md` to cover validator enablement, guard env flags, and read-only verify user. | Deploy doc updated; offline kit section mentions validator scripts; compliance checklist appended. |
|
||||
| DOCS-AOC-19-009 | DONE (2025-10-26) | Docs Guild, Authority Core | AUTH-AOC-19-001 | Update AOC docs/samples to reflect new `advisory:*`, `vex:*`, and `aoc:verify` scopes. | Docs reference new scopes, samples aligned, compliance checklist updated. |
|
||||
|
||||
## Air-Gapped Mode (Epic 16)
|
||||
| ID | Status | Owner(s) | Depends on | Description | Exit Criteria |
|
||||
|----|--------|----------|------------|-------------|---------------|
|
||||
| DOCS-AIRGAP-56-001 | TODO | Docs Guild, AirGap Controller Guild | AIRGAP-CTL-56-002 | Publish `/docs/airgap/overview.md` outlining modes, lifecycle, responsibilities, and imposed rule banner. | Doc merged; banner present; diagrams included. |
|
||||
| DOCS-AIRGAP-56-002 | TODO | Docs Guild, DevOps Guild | DEVOPS-AIRGAP-56-001 | Author `/docs/airgap/sealing-and-egress.md` covering network policies, EgressPolicy facade usage, and verification steps. | Doc merged; examples validated; banner included. |
|
||||
| DOCS-AIRGAP-56-003 | TODO | Docs Guild, Exporter Guild | EXPORT-AIRGAP-56-001 | Create `/docs/airgap/mirror-bundles.md` describing bundle format, DSSE/TUF/Merkle validation, creation/import workflows. | Doc merged; sample commands verified; banner present. |
|
||||
| DOCS-AIRGAP-56-004 | TODO | Docs Guild, Deployment Guild | DEVOPS-AIRGAP-56-003 | Publish `/docs/airgap/bootstrap.md` detailing Bootstrap Pack creation, validation, and install procedures. | Doc merged; checklist appended; screenshots verified. |
|
||||
| DOCS-AIRGAP-57-001 | TODO | Docs Guild, AirGap Time Guild | AIRGAP-TIME-58-001 | Write `/docs/airgap/staleness-and-time.md` explaining time anchors, drift policies, staleness budgets, and UI indicators. | Doc merged; math checked; banner included. |
|
||||
| DOCS-AIRGAP-57-002 | TODO | Docs Guild, Console Guild | CONSOLE-AIRGAP-57-001 | Publish `/docs/console/airgap.md` covering sealed badge, import wizard, staleness dashboards. | Doc merged; screenshots captured; banner present. |
|
||||
| DOCS-AIRGAP-57-003 | TODO | Docs Guild, CLI Guild | CLI-AIRGAP-57-001 | Publish `/docs/cli/airgap.md` documenting commands, examples, exit codes. | Doc merged; examples validated; banner present. |
|
||||
| DOCS-AIRGAP-57-004 | TODO | Docs Guild, Ops Guild | DEVOPS-AIRGAP-56-002 | Create `/docs/airgap/operations.md` with runbooks for imports, failure recovery, and auditing. | Doc merged; runbooks rehearsed; banner included. |
|
||||
| DOCS-AIRGAP-58-001 | TODO | Docs Guild, Product Guild | CONSOLE-AIRGAP-58-002 | Provide `/docs/airgap/degradation-matrix.md` enumerating feature availability, fallbacks, remediation. | Doc merged; matrix reviewed; banner included. |
|
||||
| DOCS-AIRGAP-58-002 | TODO | Docs Guild, Security Guild | PROV-OBS-54-001 | Update `/docs/security/trust-and-signing.md` with DSSE/TUF roots, rotation, and signed time tokens. | Doc merged; security sign-off recorded; banner present. |
|
||||
| DOCS-AIRGAP-58-003 | TODO | Docs Guild, DevEx Guild | AIRGAP-POL-56-001 | Publish `/docs/dev/airgap-contracts.md` describing EgressPolicy usage, sealed-mode tests, linting. | Doc merged; sample code validated; banner included. |
|
||||
| DOCS-AIRGAP-58-004 | TODO | Docs Guild, Evidence Locker Guild | EVID-OBS-55-001 | Document `/docs/airgap/portable-evidence.md` for exporting/importing portable evidence bundles across enclaves. | Doc merged; verification steps tested; banner present. |
|
||||
|
||||
## SDKs & OpenAPI (Epic 17)
|
||||
| ID | Status | Owner(s) | Depends on | Description | Exit Criteria |
|
||||
|----|--------|----------|------------|-------------|---------------|
|
||||
| DOCS-OAS-61-001 | TODO | Docs Guild, API Contracts Guild | OAS-61-002 | Publish `/docs/api/overview.md` covering auth, tenancy, pagination, idempotency, rate limits with banner. | Doc merged; examples validated; banner present. |
|
||||
| DOCS-OAS-61-002 | TODO | Docs Guild, API Governance Guild | APIGOV-61-001 | Author `/docs/api/conventions.md` capturing naming, errors, filters, sorting, examples. | Doc merged; lint passes; banner included. |
|
||||
| DOCS-OAS-61-003 | TODO | Docs Guild, API Governance Guild | APIGOV-63-001 | Publish `/docs/api/versioning.md` describing SemVer, deprecation headers, migration playbooks. | Doc merged; example headers validated; banner present. |
|
||||
| DOCS-OAS-62-001 | TODO | Docs Guild, Developer Portal Guild | DEVPORT-62-002 | Stand up `/docs/api/reference/` auto-generated site; integrate with portal nav. | Reference site builds; search works; banner included. |
|
||||
| DOCS-SDK-62-001 | TODO | Docs Guild, SDK Generator Guild | SDKGEN-63-001 | Publish `/docs/sdks/overview.md` plus language guides (`typescript.md`, `python.md`, `go.md`, `java.md`). | Docs merged; code samples pulled from tested examples; banner present. |
|
||||
| DOCS-DEVPORT-62-001 | TODO | Docs Guild, Developer Portal Guild | DEVPORT-62-001 | Document `/docs/devportal/publishing.md` for build pipeline, offline bundle steps. | Doc merged; cross-links validated; banner included. |
|
||||
| DOCS-CONTRIB-62-001 | TODO | Docs Guild, API Governance Guild | APIGOV-61-001 | Publish `/docs/contributing/api-contracts.md` detailing how to edit OAS, lint rules, compatibility checks. | Doc merged; banner present; examples validated. |
|
||||
| DOCS-TEST-62-001 | TODO | Docs Guild, Contract Testing Guild | CONTR-62-001 | Author `/docs/testing/contract-testing.md` covering mock server, replay tests, golden fixtures. | Doc merged; references to tooling validated; banner present. |
|
||||
| DOCS-SEC-62-001 | TODO | Docs Guild, Authority Core | AUTH-AIRGAP-56-001 | Update `/docs/security/auth-scopes.md` with OAuth2/PAT scopes, tenancy header usage. | Doc merged; scope tables verified; banner included. |
|
||||
| DOCS-AIRGAP-DEVPORT-64-001 | TODO | Docs Guild, DevPortal Offline Guild | DVOFF-64-001 | Create `/docs/airgap/devportal-offline.md` describing offline bundle usage and verification. | Doc merged; verification steps tested; banner present. |
|
||||
|
||||
## Risk Profiles (Epic 18)
|
||||
| ID | Status | Owner(s) | Depends on | Description | Exit Criteria |
|
||||
|----|--------|----------|------------|-------------|---------------|
|
||||
| DOCS-RISK-66-001 | TODO | Docs Guild, Risk Profile Schema Guild | POLICY-RISK-66-001 | Publish `/docs/risk/overview.md` covering concepts and glossary. | Doc merged with banner; terminology reviewed. |
|
||||
| DOCS-RISK-66-002 | TODO | Docs Guild, Policy Guild | POLICY-RISK-66-003 | Author `/docs/risk/profiles.md` (authoring, versioning, scope). | Doc merged; schema examples validated; banner present. |
|
||||
| DOCS-RISK-66-003 | TODO | Docs Guild, Risk Engine Guild | RISK-ENGINE-67-001 | Publish `/docs/risk/factors.md` cataloging signals, transforms, reducers, TTLs. | Document merged; tables verified; banner included. |
|
||||
| DOCS-RISK-66-004 | TODO | Docs Guild, Risk Engine Guild | RISK-ENGINE-66-002 | Create `/docs/risk/formulas.md` detailing math, normalization, gating, severity. | Doc merged; equations rendered; banner present. |
|
||||
| DOCS-RISK-67-001 | TODO | Docs Guild, Risk Engine Guild | RISK-ENGINE-68-001 | Publish `/docs/risk/explainability.md` showing artifact schema and UI screenshots. | Doc merged; CLI examples validated; banner included. |
|
||||
| DOCS-RISK-67-002 | TODO | Docs Guild, API Guild | POLICY-RISK-67-002 | Produce `/docs/risk/api.md` with endpoint reference/examples. | Doc merged; OAS examples synced; banner present. |
|
||||
| DOCS-RISK-67-003 | TODO | Docs Guild, Console Guild | CONSOLE-RISK-66-001 | Document `/docs/console/risk-ui.md` for authoring, simulation, dashboards. | Doc merged; screenshots updated; banner included. |
|
||||
| DOCS-RISK-67-004 | TODO | Docs Guild, CLI Guild | CLI-RISK-66-001 | Publish `/docs/cli/risk.md` covering CLI workflows. | Doc merged; command examples validated; banner present. |
|
||||
| DOCS-RISK-68-001 | TODO | Docs Guild, Export Guild | RISK-BUNDLE-69-001 | Add `/docs/airgap/risk-bundles.md` for offline factor bundles. | Doc merged; verification steps confirmed; banner included. |
|
||||
| DOCS-RISK-68-002 | TODO | Docs Guild, Security Guild | POLICY-RISK-66-003 | Update `/docs/security/aoc-invariants.md` with risk scoring provenance guarantees. | Doc merged; audit references updated; banner present. |
|
||||
|
||||
## Attestor Console (Epic 19)
|
||||
| ID | Status | Owner(s) | Depends on | Description | Exit Criteria |
|
||||
|----|--------|----------|------------|-------------|---------------|
|
||||
| DOCS-ATTEST-73-001 | TODO | Docs Guild, Attestor Service Guild | ATTEST-TYPES-73-001 | Publish `/docs/attestor/overview.md` with imposed rule banner. | Doc merged; terminology validated. |
|
||||
| DOCS-ATTEST-73-002 | TODO | Docs Guild, Attestation Payloads Guild | ATTEST-TYPES-73-002 | Write `/docs/attestor/payloads.md` with schemas/examples. | Doc merged; examples validated via tests. |
|
||||
| DOCS-ATTEST-73-003 | TODO | Docs Guild, Policy Guild | POLICY-ATTEST-73-002 | Publish `/docs/attestor/policies.md` covering verification policies. | Doc merged; policy examples validated. |
|
||||
| DOCS-ATTEST-73-004 | TODO | Docs Guild, Attestor Service Guild | ATTESTOR-73-002 | Add `/docs/attestor/workflows.md` detailing ingest, verify, bulk operations. | Doc merged; workflows tested. |
|
||||
| DOCS-ATTEST-74-001 | TODO | Docs Guild, KMS Guild | KMS-73-001 | Publish `/docs/attestor/keys-and-issuers.md`. | Doc merged; rotation guidance verified. |
|
||||
| DOCS-ATTEST-74-002 | TODO | Docs Guild, Transparency Guild | TRANSP-74-001 | Document `/docs/attestor/transparency.md` with witness usage/offline validation. | Doc merged; proofs validated. |
|
||||
| DOCS-ATTEST-74-003 | TODO | Docs Guild, Attestor Console Guild | CONSOLE-ATTEST-73-001 | Write `/docs/console/attestor-ui.md` with screenshots/workflows. | Doc merged; screenshots captured; banner present. |
|
||||
| DOCS-ATTEST-74-004 | TODO | Docs Guild, CLI Attestor Guild | CLI-ATTEST-73-001 | Publish `/docs/cli/attest.md` covering CLI usage. | Doc merged; commands validated. |
|
||||
| DOCS-ATTEST-75-001 | TODO | Docs Guild, Export Attestation Guild | EXPORT-ATTEST-75-002 | Add `/docs/attestor/airgap.md` for attestation bundles. | Doc merged; verification steps confirmed. |
|
||||
| DOCS-ATTEST-75-002 | TODO | Docs Guild, Security Guild | ATTESTOR-73-002 | Update `/docs/security/aoc-invariants.md` with attestation invariants. | Doc merged; invariants detailed. |
|
||||
## Policy Engine v2
|
||||
|
||||
| ID | Status | Owner(s) | Depends on | Description | Exit Criteria |
|
||||
|----|--------|----------|------------|-------------|---------------|
|
||||
| DOCS-POLICY-20-001 | DONE (2025-10-26) | Docs Guild, Policy Guild | POLICY-ENGINE-20-000 | Author `/docs/policy/overview.md` covering concepts, inputs/outputs, determinism, and compliance checklist. | Doc published with diagrams + glossary; lint passes; checklist included. |
|
||||
| DOCS-POLICY-20-002 | DONE (2025-10-26) | Docs Guild, Policy Guild | POLICY-ENGINE-20-001 | Write `/docs/policy/dsl.md` with grammar, built-ins, examples, anti-patterns. | DSL doc includes grammar tables, examples, compliance checklist; validated against parser tests. |
|
||||
| DOCS-POLICY-20-003 | DONE (2025-10-26) | Docs Guild, Authority Core | AUTH-POLICY-20-001 | Publish `/docs/policy/lifecycle.md` describing draft→approve workflow, roles, audit, compliance list. | Lifecycle doc linked from UI/CLI help; approvals roles documented; checklist appended. |
|
||||
| DOCS-POLICY-20-004 | DONE (2025-10-26) | Docs Guild, Scheduler Guild | SCHED-MODELS-20-001 | Create `/docs/policy/runs.md` detailing run modes, incremental mechanics, cursors, replay. | Run doc includes sequence diagrams + compliance checklist; cross-links to scheduler docs. |
|
||||
| DOCS-POLICY-20-005 | DONE (2025-10-26) | Docs Guild, BE-Base Platform Guild | WEB-POLICY-20-001 | Draft `/docs/api/policy.md` describing endpoints, schemas, error codes. | API doc validated against OpenAPI; examples included; checklist appended. |
|
||||
| DOCS-POLICY-20-006 | DONE (2025-10-26) | Docs Guild, DevEx/CLI Guild | CLI-POLICY-20-002 | Produce `/docs/cli/policy.md` with command usage, exit codes, JSON output contracts. | CLI doc includes examples, exit codes, compliance checklist. |
|
||||
| DOCS-POLICY-20-007 | DONE (2025-10-26) | Docs Guild, UI Guild | UI-POLICY-20-001 | Document `/docs/ui/policy-editor.md` covering editor, simulation, diff workflows, approvals. | UI doc includes screenshots/placeholders, accessibility notes, compliance checklist. |
|
||||
| DOCS-POLICY-20-008 | DONE (2025-10-26) | Docs Guild, Architecture Guild | POLICY-ENGINE-20-003 | Write `/docs/architecture/policy-engine.md` (new epic content) with sequence diagrams, selection strategy, schema. | Architecture doc merged with diagrams; compliance checklist appended; references updated. |
|
||||
| DOCS-POLICY-20-009 | DONE (2025-10-26) | Docs Guild, Observability Guild | POLICY-ENGINE-20-007 | Add `/docs/observability/policy.md` for metrics/traces/logs, sample dashboards. | Observability doc includes metrics tables, dashboard screenshots, checklist. |
|
||||
| DOCS-POLICY-20-010 | DONE (2025-10-26) | Docs Guild, Security Guild | AUTH-POLICY-20-002 | Publish `/docs/security/policy-governance.md` covering scopes, approvals, tenancy, least privilege. | Security doc merged; compliance checklist appended; reviewed by Security Guild. |
|
||||
| DOCS-POLICY-20-011 | DONE (2025-10-26) | Docs Guild, Policy Guild | POLICY-ENGINE-20-001 | Populate `/docs/examples/policies/` with baseline/serverless/internal-only samples and commentary. | Example policies committed with explanations; lint passes; compliance checklist per file. |
|
||||
| DOCS-POLICY-20-012 | DONE (2025-10-26) | Docs Guild, Support Guild | WEB-POLICY-20-003 | Draft `/docs/faq/policy-faq.md` addressing common pitfalls, VEX conflicts, determinism issues. | FAQ published with Q/A entries, cross-links, compliance checklist. |
|
||||
|
||||
## Graph Explorer v1
|
||||
|
||||
| ID | Status | Owner(s) | Depends on | Description | Exit Criteria |
|
||||
|----|--------|----------|------------|-------------|---------------|
|
||||
|
||||
## Link-Not-Merge v1
|
||||
|
||||
| ID | Status | Owner(s) | Depends on | Description | Exit Criteria |
|
||||
|----|--------|----------|------------|-------------|---------------|
|
||||
| DOCS-LNM-22-001 | BLOCKED (2025-10-27) | Docs Guild, Concelier Guild | CONCELIER-LNM-21-001..003 | Author `/docs/advisories/aggregation.md` covering observation vs linkset, conflict handling, AOC requirements, and reviewer checklist. | Draft doc merged with examples + checklist; final sign-off blocked until Concelier schema/API tasks land. |
|
||||
> Blocker (2025-10-27): `CONCELIER-LNM-21-001..003` still TODO; update doc + fixtures once schema/API implementations are available.
|
||||
| DOCS-LNM-22-002 | BLOCKED (2025-10-27) | Docs Guild, Excititor Guild | EXCITITOR-LNM-21-001..003 | Publish `/docs/vex/aggregation.md` describing VEX observation/linkset model, product matching, conflicts. | Draft doc merged with fixtures; final approval blocked until Excititor observation/linkset work ships. |
|
||||
> Blocker (2025-10-27): `EXCITITOR-LNM-21-001..003` remain TODO; refresh doc, fixtures, and examples post-implementation.
|
||||
| DOCS-LNM-22-003 | BLOCKED (2025-10-27) | Docs Guild, BE-Base Platform Guild | WEB-LNM-21-001..003 | Update `/docs/api/advisories.md` and `/docs/api/vex.md` for new endpoints, parameters, errors, exports. | Draft pending gateway/API delivery; unblock once endpoints + OpenAPI specs are available. |
|
||||
> Blocker (2025-10-27): `WEB-LNM-21-001..003` all TODO—no gateway endpoints/OpenAPI to document yet.
|
||||
| DOCS-LNM-22-004 | TODO | Docs Guild, Policy Guild | POLICY-ENGINE-40-001 | Create `/docs/policy/effective-severity.md` detailing severity selection strategies from multiple sources. | Doc merged with policy examples; checklist included. |
|
||||
| DOCS-LNM-22-005 | BLOCKED (2025-10-27) | Docs Guild, UI Guild | UI-LNM-22-001..003 | Document `/docs/ui/evidence-panel.md` with screenshots, conflict badges, accessibility guidance. | Awaiting UI implementation to capture screenshots + flows; unblock once Evidence panel ships. |
|
||||
> Blocker (2025-10-27): `UI-LNM-22-001..003` all TODO; documentation requires final UI states and accessibility audit artifacts.
|
||||
|
||||
## StellaOps Console (Sprint 23)
|
||||
|
||||
| ID | Status | Owner(s) | Depends on | Description | Exit Criteria |
|
||||
|----|--------|----------|------------|-------------|---------------|
|
||||
| DOCS-CONSOLE-23-001 | DONE (2025-10-26) | Docs Guild, Console Guild | CONSOLE-CORE-23-004 | Publish `/docs/ui/console-overview.md` covering IA, tenant model, global filters, and AOC alignment with compliance checklist. | Doc merged with diagrams + overview tables; checklist appended; Console Guild sign-off. |
|
||||
| DOCS-CONSOLE-23-002 | DONE (2025-10-26) | Docs Guild, Console Guild | DOCS-CONSOLE-23-001 | Author `/docs/ui/navigation.md` detailing routes, breadcrumbs, keyboard shortcuts, deep links, and tenant context switching. | Navigation doc merged with shortcut tables and screenshots; accessibility checklist satisfied. |
|
||||
| DOCS-CONSOLE-23-003 | DONE (2025-10-26) | Docs Guild, SBOM Service Guild, Console Guild | SBOM-CONSOLE-23-001, CONSOLE-FEAT-23-102 | Document `/docs/ui/sbom-explorer.md` (catalog, detail, graph overlays, exports) including compliance checklist and performance tips. | Doc merged with annotated screenshots, export instructions, and overlay examples; checklist appended. |
|
||||
| DOCS-CONSOLE-23-004 | DONE (2025-10-26) | Docs Guild, Concelier Guild, Excititor Guild | CONCELIER-CONSOLE-23-001, EXCITITOR-CONSOLE-23-001 | Produce `/docs/ui/advisories-and-vex.md` explaining aggregation-not-merge, conflict indicators, raw viewers, and provenance banners. | Doc merged; raw JSON examples included; compliance checklist complete. |
|
||||
| DOCS-CONSOLE-23-005 | DONE (2025-10-26) | Docs Guild, Policy Guild | POLICY-CONSOLE-23-001, CONSOLE-FEAT-23-104 | Write `/docs/ui/findings.md` describing filters, saved views, explain drawer, exports, and CLI parity callouts. | Doc merged with filter matrix + explain walkthrough; checklist appended. |
|
||||
| DOCS-CONSOLE-23-006 | DONE (2025-10-26) | Docs Guild, Policy Guild, Product Ops | POLICY-CONSOLE-23-002, CONSOLE-FEAT-23-105 | Publish `/docs/ui/policies.md` with editor, simulation, approvals, compliance checklist, and RBAC mapping. | Doc merged; Monaco screenshots + simulation diff examples included; approval flow described; checklist appended. |
|
||||
| DOCS-CONSOLE-23-007 | DONE (2025-10-26) | Docs Guild, Scheduler Guild | SCHED-CONSOLE-23-001, CONSOLE-FEAT-23-106 | Document `/docs/ui/runs.md` covering queues, live progress, diffs, retries, evidence downloads, and troubleshooting. | Doc merged with SSE troubleshooting, metrics references, compliance checklist. |
|
||||
| DOCS-CONSOLE-23-008 | DONE (2025-10-26) | Docs Guild, Authority Guild | AUTH-CONSOLE-23-002, CONSOLE-FEAT-23-108 | Draft `/docs/ui/admin.md` describing users/roles, tenants, tokens, integrations, fresh-auth prompts, and RBAC mapping. | Doc merged with tables for scopes vs roles, screenshots, compliance checklist. |
|
||||
| DOCS-CONSOLE-23-009 | DONE (2025-10-27) | Docs Guild, DevOps Guild | DOWNLOADS-CONSOLE-23-001, CONSOLE-FEAT-23-109 | Publish `/docs/ui/downloads.md` listing product images, commands, offline instructions, parity with CLI, and compliance checklist. | Doc merged; manifest sample included; copy-to-clipboard guidance documented; checklist complete. |
|
||||
| DOCS-CONSOLE-23-010 | DONE (2025-10-27) | Docs Guild, Deployment Guild, Console Guild | DEVOPS-CONSOLE-23-002, CONSOLE-REL-23-301 | Write `/docs/deploy/console.md` (Helm, ingress, TLS, CSP, env vars, health checks) with compliance checklist. | Deploy doc merged; templates validated; CSP guidance included; checklist appended. |
|
||||
| DOCS-CONSOLE-23-011 | DONE (2025-10-28) | Docs Guild, Deployment Guild | DOCS-CONSOLE-23-010 | Update `/docs/install/docker.md` to cover Console image, Compose/Helm usage, offline tarballs, parity with CLI. | Doc updated with new sections; commands validated; compliance checklist appended. |
|
||||
| DOCS-CONSOLE-23-012 | DONE (2025-10-28) | Docs Guild, Security Guild | AUTH-CONSOLE-23-003, WEB-CONSOLE-23-002 | Publish `/docs/security/console-security.md` detailing OIDC flows, scopes, CSP, fresh-auth, evidence handling, and compliance checklist. | Security doc merged; threat model notes included; checklist appended. |
|
||||
| DOCS-CONSOLE-23-013 | DONE (2025-10-28) | Docs Guild, Observability Guild | TELEMETRY-CONSOLE-23-001, CONSOLE-QA-23-403 | Write `/docs/observability/ui-telemetry.md` cataloguing metrics/logs/traces, dashboards, alerts, and feature flags. | Doc merged with instrumentation tables, dashboard screenshots, checklist appended. |
|
||||
| DOCS-CONSOLE-23-014 | DONE (2025-10-28) | Docs Guild, Console Guild, CLI Guild | CONSOLE-DOC-23-502 | Maintain `/docs/cli-vs-ui-parity.md` matrix and integrate CI check guidance. | Matrix published with parity status, CI workflow documented, compliance checklist appended. |
|
||||
> 2025-10-28: Install Docker guide references pending CLI commands (`stella downloads manifest`, `stella downloads mirror`, `stella console status`). Update once CLI parity lands.
|
||||
| DOCS-CONSOLE-23-015 | DONE (2025-10-27) | Docs Guild, Architecture Guild | CONSOLE-CORE-23-001, WEB-CONSOLE-23-001 | Produce `/docs/architecture/console.md` describing frontend packages, data flow diagrams, SSE design, performance budgets. | Architecture doc merged with diagrams + compliance checklist; reviewers approve. |
|
||||
| DOCS-CONSOLE-23-016 | DONE (2025-10-28) | Docs Guild, Accessibility Guild | CONSOLE-QA-23-402, CONSOLE-FEAT-23-102 | Refresh `/docs/accessibility.md` with Console-specific keyboard flows, color tokens, testing tools, and compliance checklist updates. | Accessibility doc updated; audits referenced; checklist appended. |
|
||||
> 2025-10-28: Added guide covering keyboard matrix, screen reader behaviour, colour/focus tokens, testing workflow, offline guidance, and compliance checklist.
|
||||
| DOCS-CONSOLE-23-017 | DONE (2025-10-27) | Docs Guild, Console Guild | CONSOLE-FEAT-23-101..109 | Create `/docs/examples/ui-tours.md` providing triage, audit, policy rollout walkthroughs with annotated screenshots and GIFs. | UI tours doc merged; capture instructions + asset placeholders committed; compliance checklist appended. |
|
||||
| DOCS-CONSOLE-23-018 | DONE (2025-10-27) | Docs Guild, Security Guild | DOCS-CONSOLE-23-012 | Execute console security compliance checklist and capture Security Guild sign-off in Sprint 23 log. | Checklist completed; findings addressed or tickets filed; sign-off noted in updates file. |
|
||||
| DOCS-LNM-22-006 | DONE (2025-10-27) | Docs Guild, Architecture Guild | CONCELIER-LNM-21-001..005, EXCITITOR-LNM-21-001..005 | Refresh `/docs/architecture/conseiller.md` and `/docs/architecture/excitator.md` describing observation/linkset pipelines and event contracts. | Architecture docs updated with observation/linkset flow + event tables; revisit once service implementations land. |
|
||||
> Follow-up: align diagrams/examples after `CONCELIER-LNM-21` & `EXCITITOR-LNM-21` work merges (currently TODO).
|
||||
| DOCS-LNM-22-007 | TODO | Docs Guild, Observability Guild | CONCELIER-LNM-21-005, EXCITITOR-LNM-21-005, DEVOPS-LNM-22-002 | Publish `/docs/observability/aggregation.md` with metrics/traces/logs/SLOs. | Observability doc merged; dashboards referenced; checklist appended. |
|
||||
| DOCS-LNM-22-008 | TODO | Docs Guild, DevOps Guild | MERGE-LNM-21-001, CONCELIER-LNM-21-102 | Write `/docs/migration/no-merge.md` describing migration plan, backfill steps, rollback, feature flags. | Migration doc approved by stakeholders; checklist appended. |
|
||||
|
||||
## Policy Engine + Editor v1
|
||||
|
||||
| ID | Status | Owner(s) | Depends on | Description | Exit Criteria |
|
||||
|----|--------|----------|------------|-------------|---------------|
|
||||
| DOCS-POLICY-23-001 | TODO | Docs Guild, Policy Guild | POLICY-SPL-23-001..003 | Author `/docs/policy/overview.md` describing SPL philosophy, layering, and glossary with reviewer checklist. | Doc merged; lint passes; checklist appended. |
|
||||
| DOCS-POLICY-23-002 | TODO | Docs Guild, Policy Guild | POLICY-SPL-23-001 | Write `/docs/policy/spl-v1.md` (language reference, JSON Schema, examples). | Reference published with schema snippets; checklist completed. |
|
||||
| DOCS-POLICY-23-003 | TODO | Docs Guild, Policy Guild | POLICY-ENGINE-50-001..004 | Produce `/docs/policy/runtime.md` covering compiler, evaluator, caching, events, SLOs. | Runtime doc merged with diagrams; observability references included. |
|
||||
| DOCS-POLICY-23-004 | TODO | Docs Guild, UI Guild | UI-POLICY-23-001..006 | Document `/docs/policy/editor.md` (UI walkthrough, validation, simulation, approvals). | Editor doc merged with screenshots; accessibility checklist satisfied. |
|
||||
| DOCS-POLICY-23-005 | TODO | Docs Guild, Security Guild | AUTH-POLICY-23-001..002 | Publish `/docs/policy/governance.md` (roles, scopes, approvals, signing, exceptions). | Governance doc merged; checklist appended. |
|
||||
| DOCS-POLICY-23-006 | TODO | Docs Guild, BE-Base Platform Guild | WEB-POLICY-23-001..004 | Update `/docs/api/policy.md` with new endpoints, schemas, errors, pagination. | API doc aligns with OpenAPI; examples validated; checklist included. |
|
||||
| DOCS-POLICY-23-007 | TODO | Docs Guild, DevEx/CLI Guild | CLI-POLICY-23-004..006 | Update `/docs/cli/policy.md` for lint/simulate/activate/history commands, exit codes. | CLI doc updated; samples verified; checklist appended. |
|
||||
| DOCS-POLICY-23-008 | TODO | Docs Guild, Architecture Guild | POLICY-ENGINE-50-005..006 | Refresh `/docs/architecture/policy-engine.md` with data model, sequence diagrams, event flows. | Architecture doc merged with diagrams; checklist appended. |
|
||||
| DOCS-POLICY-23-009 | TODO | Docs Guild, DevOps Guild | MERGE-LNM-21-001, DEVOPS-LNM-22-001 | Create `/docs/migration/policy-parity.md` covering dual-run parity plan and rollback. | Migration doc approved; checklist appended. |
|
||||
| DOCS-POLICY-23-010 | TODO | Docs Guild, UI Guild | UI-POLICY-23-006 | Write `/docs/ui/explainers.md` showing explain trees, evidence overlays, interpretation guidance. | Doc merged with annotated screenshots; checklist appended. |
|
||||
|
||||
## Graph & Vuln Explorer v1
|
||||
|
||||
| ID | Status | Owner(s) | Depends on | Description | Exit Criteria |
|
||||
|----|--------|----------|------------|-------------|---------------|
|
||||
| DOCS-GRAPH-24-001 | TODO | Docs Guild, UI Guild | UI-GRAPH-24-001..006 | Author `/docs/ui/sbom-graph-explorer.md` detailing overlays, filters, saved views, accessibility, and AOC visibility. | Doc merged; screenshots included; checklist appended. |
|
||||
| DOCS-GRAPH-24-002 | TODO | Docs Guild, UI Guild | UI-GRAPH-24-005 | Publish `/docs/ui/vulnerability-explorer.md` covering table usage, grouping, fix suggestions, Why drawer. | Doc merged with annotated images; accessibility checklist satisfied. |
|
||||
| DOCS-GRAPH-24-003 | TODO | Docs Guild, SBOM Service Guild | SBOM-GRAPH-24-001..003 | Create `/docs/architecture/graph-index.md` describing data model, ingestion pipeline, caches, events. | Architecture doc merged with diagrams; checklist appended. |
|
||||
| DOCS-GRAPH-24-004 | TODO | Docs Guild, BE-Base Platform Guild | WEB-GRAPH-24-001..003 | Document `/docs/api/graph.md` and `/docs/api/vuln.md` avec endpoints, parameters, errors, RBAC. | API docs aligned with OpenAPI; examples validated; checklist appended. |
|
||||
| DOCS-GRAPH-24-005 | TODO | Docs Guild, DevEx/CLI Guild | CLI-GRAPH-24-001..003 | Update `/docs/cli/graph-and-vuln.md` covering new CLI commands, exit codes, scripting. | CLI doc merged; examples tested; checklist appended. |
|
||||
| DOCS-GRAPH-24-006 | TODO | Docs Guild, Policy Guild | POLICY-ENGINE-60-001..002 | Write `/docs/policy/ui-integration.md` explaining overlays, cache usage, simulator contracts. | Doc merged; references cross-linked; checklist appended. |
|
||||
| DOCS-GRAPH-24-007 | TODO | Docs Guild, DevOps Guild | DEVOPS-GRAPH-24-001..003 | Produce `/docs/migration/graph-parity.md` with rollout plan, parity checks, fallback guidance. | Migration doc approved; checklist appended. |
|
||||
|
||||
## Exceptions v1
|
||||
|
||||
| ID | Status | Owner(s) | Depends on | Description | Exit Criteria |
|
||||
|----|--------|----------|------------|-------------|---------------|
|
||||
| DOCS-EXC-25-001 | TODO | Docs Guild, Governance Guild | WEB-EXC-25-001 | Author `/docs/governance/exceptions.md` covering lifecycle, scope patterns, examples, compliance checklist. | Doc merged; reviewers sign off; checklist included. |
|
||||
| DOCS-EXC-25-002 | TODO | Docs Guild, Authority Core | AUTH-EXC-25-001 | Publish `/docs/governance/approvals-and-routing.md` detailing roles, routing matrix, MFA rules, audit trails. | Doc merged; routing examples validated; checklist appended. |
|
||||
| DOCS-EXC-25-003 | TODO | Docs Guild, BE-Base Platform Guild | WEB-EXC-25-001..003 | Create `/docs/api/exceptions.md` with endpoints, payloads, errors, idempotency notes. | API doc aligned with OpenAPI; examples tested; checklist appended. |
|
||||
| DOCS-EXC-25-004 | DONE (2025-10-27) | Docs Guild, Policy Guild | POLICY-ENGINE-70-001 | Document `/docs/policy/exception-effects.md` explaining evaluation order, conflicts, simulation. | Doc merged; tests cross-referenced; checklist appended. |
|
||||
| DOCS-EXC-25-005 | TODO | Docs Guild, UI Guild | UI-EXC-25-001..004 | Write `/docs/ui/exception-center.md` with UI walkthrough, badges, accessibility, shortcuts. | Doc merged with screenshots; accessibility checklist completed. |
|
||||
| DOCS-EXC-25-006 | TODO | Docs Guild, DevEx/CLI Guild | CLI-EXC-25-001..002 | Update `/docs/cli/exceptions.md` covering command usage and exit codes. | CLI doc updated; examples validated; checklist appended. |
|
||||
| DOCS-EXC-25-007 | TODO | Docs Guild, DevOps Guild | SCHED-WORKER-25-101, DEVOPS-GRAPH-24-003 | Publish `/docs/migration/exception-governance.md` describing cutover from legacy suppressions, notifications, rollback. | Migration doc approved; checklist included. |
|
||||
|
||||
> Update statuses (TODO/DOING/REVIEW/DONE/BLOCKED) as progress changes. Keep guides in sync with configuration samples under `etc/`.
|
||||
|
||||
> Remark (2025-10-13, DOC4.AUTH-PDG): Rate limit guide published (`docs/security/rate-limits.md`) and handed to plugin docs team for diagram uplift once PLG6.DIAGRAM lands.
|
||||
|
||||
## Orchestrator Dashboard (Epic 9)
|
||||
|
||||
| ID | Status | Owner(s) | Depends on | Description | Exit Criteria |
|
||||
|----|--------|----------|------------|-------------|---------------|
|
||||
| DOCS-ORCH-32-001 | TODO | Docs Guild | ORCH-SVC-32-001, AUTH-ORCH-32-001 | Author `/docs/orchestrator/overview.md` covering mission, roles, AOC alignment, governance, with imposed rule reminder. | Doc merged with diagrams; imposed rule statement included; entry linked from docs index. |
|
||||
| DOCS-ORCH-32-002 | TODO | Docs Guild | ORCH-SVC-32-002 | Author `/docs/orchestrator/architecture.md` detailing scheduler, DAGs, rate limits, data model, message bus, storage layout, restating imposed rule. | Architecture doc merged; diagrams reviewed; imposed rule noted. |
|
||||
| DOCS-ORCH-33-001 | TODO | Docs Guild | ORCH-SVC-33-001..004, WEB-ORCH-33-001 | Publish `/docs/orchestrator/api.md` (REST/WebSocket endpoints, payloads, error codes) with imposed rule note. | API doc merged; examples validated; imposed rule appended. |
|
||||
| DOCS-ORCH-33-002 | TODO | Docs Guild | CONSOLE-ORCH-32-002, CONSOLE-ORCH-33-001..002 | Publish `/docs/orchestrator/console.md` covering screens, a11y, live updates, control actions, reiterating imposed rule. | Console doc merged with screenshots; accessibility checklist done; imposed rule statement present. |
|
||||
| DOCS-ORCH-33-003 | TODO | Docs Guild | CLI-ORCH-33-001 | Publish `/docs/orchestrator/cli.md` documenting commands, options, exit codes, streaming output, offline usage, and imposed rule. | CLI doc merged; examples tested; imposed rule appended. |
|
||||
| DOCS-ORCH-34-001 | TODO | Docs Guild | ORCH-SVC-34-002, LEDGER-34-101 | Author `/docs/orchestrator/run-ledger.md` covering ledger schema, provenance chain, audit workflows, with imposed rule reminder. | Run-ledger doc merged; payload samples validated; imposed rule included; cross-links added. |
|
||||
| DOCS-ORCH-34-002 | TODO | Docs Guild | AUTH-ORCH-32-001, AUTH-ORCH-34-001 | Update `/docs/security/secrets-handling.md` for orchestrator KMS refs, redaction badges, operator hygiene, reiterating imposed rule. | Security doc merged; checklists updated; imposed rule restated; references from Console/CLI docs added. |
|
||||
| DOCS-ORCH-34-003 | TODO | Docs Guild | ORCH-SVC-33-003, ORCH-SVC-34-001, DEVOPS-ORCH-34-001 | Publish `/docs/operations/orchestrator-runbook.md` (incident playbook, backfill guide, circuit breakers, throttling) with imposed rule statement. | Runbook merged; steps validated with DevOps; imposed rule included; runbook linked from ops index. |
|
||||
| DOCS-ORCH-34-004 | TODO | Docs Guild | ORCH-SVC-32-005, WORKER-GO-33-001, WORKER-PY-33-001 | Document `/docs/schemas/artifacts.md` describing artifact kinds, schema versions, hashing, storage layout, restating imposed rule. | Schema doc merged; JSON schema provided; imposed rule included; sample payload validated. |
|
||||
| DOCS-ORCH-34-005 | TODO | Docs Guild | ORCH-SVC-34-001, DEVOPS-ORCH-34-001 | Author `/docs/slo/orchestrator-slo.md` defining SLOs, burn alerts, measurement, and reiterating imposed rule. | SLO doc merged; dashboard screenshots embedded; imposed rule appended; alerts documented. |
|
||||
|
||||
## Export Center (Epic 10)
|
||||
|
||||
| ID | Status | Owner(s) | Depends on | Description | Exit Criteria |
|
||||
|----|--------|----------|------------|-------------|---------------|
|
||||
| DOCS-EXPORT-35-001 | DONE (2025-10-29) | Docs Guild | EXPORT-SVC-35-001..006 | Author `/docs/export-center/overview.md` covering purpose, profiles, security, AOC alignment, surfaces, ending with imposed rule statement. | Doc merged with diagrams/examples; imposed rule line present; index updated. |
|
||||
| DOCS-EXPORT-35-002 | DONE (2025-10-29) | Docs Guild | EXPORT-SVC-35-002..005 | Publish `/docs/export-center/architecture.md` describing planner, adapters, manifests, signing, distribution flows, restating imposed rule. | Architecture doc merged; sequence diagrams included; rule statement appended. |
|
||||
| DOCS-EXPORT-35-003 | DONE (2025-10-29) | Docs Guild | EXPORT-SVC-35-003..004 | Publish `/docs/export-center/profiles.md` detailing schema fields, examples, compatibility, and imposed rule reminder. | Profiles doc merged; JSON schemas linked; imposed rule noted. |
|
||||
| DOCS-EXPORT-36-004 | DONE (2025-10-29) | Docs Guild | EXPORT-SVC-36-001..004, WEB-EXPORT-36-001 | Publish `/docs/export-center/api.md` covering endpoints, payloads, errors, and mention imposed rule. | API doc merged; examples validated; rule included. |
|
||||
| DOCS-EXPORT-36-005 | DONE (2025-10-29) | Docs Guild | CLI-EXPORT-35-001, CLI-EXPORT-36-001 | Publish `/docs/export-center/cli.md` with command reference, CI scripts, verification steps, restating imposed rule. | CLI doc merged; script snippets tested; rule appended. |
|
||||
| DOCS-EXPORT-36-006 | DONE (2025-10-29) | Docs Guild | EXPORT-SVC-36-001, DEVOPS-EXPORT-36-001 | Publish `/docs/export-center/trivy-adapter.md` covering field mappings, compatibility matrix, and imposed rule reminder. | Doc merged; mapping tables validated; rule included. |
|
||||
| DOCS-EXPORT-37-001 | DONE (2025-10-29) | Docs Guild | EXPORT-SVC-37-001, DEVOPS-EXPORT-37-001 | Publish `/docs/export-center/mirror-bundles.md` describing filesystem/OCI layouts, delta/encryption, import guide, ending with imposed rule. | Doc merged; diagrams provided; verification steps tested; rule stated. |
|
||||
| DOCS-EXPORT-37-002 | DONE (2025-10-29) | Docs Guild | EXPORT-SVC-35-005, EXPORT-SVC-37-002 | Publish `/docs/export-center/provenance-and-signing.md` detailing manifests, attestation flow, verification, reiterating imposed rule. | Doc merged; signature examples validated; rule appended. |
|
||||
| DOCS-EXPORT-37-003 | DONE (2025-10-29) | Docs Guild | DEVOPS-EXPORT-37-001 | Publish `/docs/operations/export-runbook.md` covering failures, tuning, capacity planning, with imposed rule reminder. | Runbook merged; procedures validated; rule included. |
|
||||
| DOCS-EXPORT-37-004 | TODO | Docs Guild | AUTH-EXPORT-37-001, EXPORT-SVC-37-002 | Publish `/docs/security/export-hardening.md` outlining RBAC, tenancy, encryption, redaction, restating imposed rule. | Security doc merged; checklist updated; rule appended. |
|
||||
| DOCS-EXPORT-37-101 | TODO | Docs Guild, DevEx/CLI Guild | CLI-EXPORT-37-001 | Refresh CLI verification sections once `stella export verify` lands (flags, exit codes, samples). | `docs/export-center/cli.md` & `docs/export-center/provenance-and-signing.md` updated with final command syntax; examples tested; rule reminder retained. |
|
||||
| DOCS-EXPORT-37-102 | TODO | Docs Guild, DevOps Guild | DEVOPS-EXPORT-37-001 | Embed export dashboards/alerts references into provenance/runbook docs after Grafana work ships. | Docs updated with dashboard IDs/alert notes; update logged; rule reminder present. |
|
||||
| DOCS-EXPORT-37-005 | TODO | Docs Guild, Exporter Service Guild | EXPORT-SVC-35-006, DEVOPS-EXPORT-36-001 | Validate Export Center docs against live Trivy/mirror bundles once implementation lands; refresh examples and CLI snippets accordingly. | Real bundle examples recorded; docs updated; verification steps confirmed with production artefacts. |
|
||||
> Note (2025-10-29): Blocked until exporter API (`EXPORT-SVC-35-006`) and Trivy/mirror adapters (`EXPORT-SVC-36-001`, `EXPORT-SVC-37-001`) ship. Requires access to CI smoke outputs (`DEVOPS-EXPORT-36-001`) for verification artifacts.
|
||||
|
||||
## Reachability v1
|
||||
|
||||
| ID | Status | Owner(s) | Depends on | Description | Exit Criteria |
|
||||
|----|--------|----------|------------|-------------|---------------|
|
||||
| DOCS-SIG-26-001 | TODO | Docs Guild, Signals Guild | SIGNALS-24-004 | Write `/docs/signals/reachability.md` covering states, scores, provenance, retention. | Doc merged with diagrams/examples; checklist appended. |
|
||||
| DOCS-SIG-26-002 | TODO | Docs Guild, Signals Guild | SIGNALS-24-002 | Publish `/docs/signals/callgraph-formats.md` with schemas and validation errors. | Doc merged; examples tested; checklist included. |
|
||||
| DOCS-SIG-26-003 | TODO | Docs Guild, Runtime Guild | SIGNALS-24-003 | Create `/docs/signals/runtime-facts.md` detailing agent capabilities, privacy safeguards, opt-in flags. | Doc merged; privacy review done; checklist appended. |
|
||||
| DOCS-SIG-26-004 | TODO | Docs Guild, Policy Guild | POLICY-ENGINE-80-001 | Document `/docs/policy/signals-weighting.md` for SPL predicates and weighting strategies. | Doc merged; sample policies validated; checklist appended. |
|
||||
| DOCS-SIG-26-005 | TODO | Docs Guild, UI Guild | UI-SIG-26-001..003 | Draft `/docs/ui/reachability-overlays.md` with badges, timelines, shortcuts. | Doc merged with screenshots; accessibility checklist completed. |
|
||||
| DOCS-SIG-26-006 | TODO | Docs Guild, DevEx/CLI Guild | CLI-SIG-26-001..002 | Update `/docs/cli/reachability.md` for new commands and automation recipes. | Doc merged; examples verified; checklist appended. |
|
||||
| DOCS-SIG-26-007 | TODO | Docs Guild, BE-Base Platform Guild | WEB-SIG-26-001..003 | Publish `/docs/api/signals.md` covering endpoints, payloads, ETags, errors. | API doc aligned with OpenAPI; examples tested; checklist appended. |
|
||||
| DOCS-SIG-26-008 | TODO | Docs Guild, DevOps Guild | DEVOPS-SIG-26-001..002 | Write `/docs/migration/enable-reachability.md` guiding rollout, fallbacks, monitoring. | Migration doc approved; checklist appended. |
|
||||
|
||||
## Policy Studio (Sprint 27)
|
||||
|
||||
| ID | Status | Owner(s) | Depends on | Description | Exit Criteria |
|
||||
|----|--------|----------|------------|-------------|---------------|
|
||||
| DOCS-POLICY-27-001 | BLOCKED (2025-10-27) | Docs Guild, Policy Guild | REGISTRY-API-27-001, POLICY-ENGINE-27-001 | Publish `/docs/policy/studio-overview.md` covering lifecycle, roles, glossary, and compliance checklist. | Doc merged with diagrams + lifecycle table; checklist appended; stakeholders sign off. |
|
||||
> Blocked by `REGISTRY-API-27-001` and `POLICY-ENGINE-27-001`; need spec + compile data.
|
||||
> Blocker: Registry OpenAPI (`REGISTRY-API-27-001`) and policy compile enrichments (`POLICY-ENGINE-27-001`) are still TODO; need final interfaces before drafting overview.
|
||||
| DOCS-POLICY-27-002 | BLOCKED (2025-10-27) | Docs Guild, Console Guild | CONSOLE-STUDIO-27-001 | Write `/docs/policy/authoring.md` detailing workspace templates, snippets, lint rules, IDE shortcuts, and best practices. | Authoring doc includes annotated screenshots, snippet catalog, compliance checklist. |
|
||||
> Blocked by `CONSOLE-STUDIO-27-001` Studio authoring UI pending.
|
||||
> Blocker: Console Studio authoring UI (`CONSOLE-STUDIO-27-001`) not implemented; awaiting UX to capture flows/snippets.
|
||||
| DOCS-POLICY-27-003 | BLOCKED (2025-10-27) | Docs Guild, Policy Registry Guild | REGISTRY-API-27-007 | Document `/docs/policy/versioning-and-publishing.md` (semver rules, attestations, rollback) with compliance checklist. | Doc merged with flow diagrams; attestation steps documented; checklist appended. |
|
||||
> Blocked by `REGISTRY-API-27-007` publish/sign pipeline outstanding.
|
||||
> Blocker: Registry publish/sign workflow (`REGISTRY-API-27-007`) pending.
|
||||
| DOCS-POLICY-27-004 | BLOCKED (2025-10-27) | Docs Guild, Scheduler Guild | REGISTRY-API-27-005, SCHED-WORKER-27-301 | Write `/docs/policy/simulation.md` covering quick vs batch sim, thresholds, evidence bundles, CLI examples. | Simulation doc includes charts, sample manifests, checklist appended. |
|
||||
> Blocked by `REGISTRY-API-27-005`/`SCHED-WORKER-27-301` batch simulation not ready.
|
||||
> Blocker: Batch simulation APIs/workers (`REGISTRY-API-27-005`, `SCHED-WORKER-27-301`) still TODO.
|
||||
| DOCS-POLICY-27-005 | BLOCKED (2025-10-27) | Docs Guild, Product Ops | REGISTRY-API-27-006 | Publish `/docs/policy/review-and-approval.md` with approver requirements, comments, webhooks, audit trail guidance. | Doc merged with role matrix + webhook schema; checklist appended. |
|
||||
> Blocked by `REGISTRY-API-27-006` review workflow not implemented.
|
||||
> Blocker: Review workflow (`REGISTRY-API-27-006`) not landed.
|
||||
| DOCS-POLICY-27-006 | BLOCKED (2025-10-27) | Docs Guild, Policy Guild | REGISTRY-API-27-008 | Author `/docs/policy/promotion.md` covering environments, canary, rollback, and monitoring steps. | Promotion doc includes examples + checklist; verified by Policy Ops. |
|
||||
> Blocked by `REGISTRY-API-27-008` promotion APIs pending.
|
||||
> Blocker: Promotion/canary APIs (`REGISTRY-API-27-008`) outstanding.
|
||||
| DOCS-POLICY-27-007 | BLOCKED (2025-10-27) | Docs Guild, DevEx/CLI Guild | CLI-POLICY-27-001..004 | Update `/docs/policy/cli.md` with new commands, JSON schemas, CI usage, and compliance checklist. | CLI doc merged with transcripts; schema references validated; checklist appended. |
|
||||
> Blocked by `CLI-POLICY-27-001..004` CLI commands missing.
|
||||
> Blocker: Policy CLI commands (`CLI-POLICY-27-001..004`) yet to implement.
|
||||
| DOCS-POLICY-27-008 | BLOCKED (2025-10-27) | Docs Guild, Policy Registry Guild | REGISTRY-API-27-001..008 | Publish `/docs/policy/api.md` describing Registry endpoints, request/response schemas, errors, and feature flags. | API doc aligned with OpenAPI; examples validated; checklist appended. |
|
||||
> Blocked by `REGISTRY-API-27-001..008` OpenAPI + endpoints incomplete.
|
||||
> Blocker: Registry OpenAPI/spec suite (`REGISTRY-API-27-001..008`) incomplete.
|
||||
| DOCS-POLICY-27-009 | BLOCKED (2025-10-27) | Docs Guild, Security Guild | AUTH-POLICY-27-002 | Create `/docs/security/policy-attestations.md` covering signing, verification, key rotation, and compliance checklist. | Security doc approved by Security Guild; verifier steps documented; checklist appended. |
|
||||
> Blocked by `AUTH-POLICY-27-002` signing enforcement pending.
|
||||
> Blocker: Authority signing enforcement (`AUTH-POLICY-27-002`) pending.
|
||||
| DOCS-POLICY-27-010 | BLOCKED (2025-10-27) | Docs Guild, Architecture Guild | REGISTRY-API-27-001, SCHED-WORKER-27-301 | Author `/docs/architecture/policy-registry.md` (service design, schemas, queues, failure modes) with diagrams and checklist. | Architecture doc merged; diagrams committed; checklist appended. |
|
||||
> Blocked by `REGISTRY-API-27-001` & `SCHED-WORKER-27-301` need delivery.
|
||||
> Blocker: Policy Registry schema/workers not delivered (see `REGISTRY-API-27-001`, `SCHED-WORKER-27-301`).
|
||||
| DOCS-POLICY-27-011 | BLOCKED (2025-10-27) | Docs Guild, Observability Guild | DEVOPS-POLICY-27-004 | Publish `/docs/observability/policy-telemetry.md` with metrics/log tables, dashboards, alerts, and compliance checklist. | Observability doc merged; dashboards linked; checklist appended. |
|
||||
> Blocked by `DEVOPS-POLICY-27-004` observability dashboards outstanding.
|
||||
> Blocker: Observability dashboards (`DEVOPS-POLICY-27-004`) not built.
|
||||
| DOCS-POLICY-27-012 | BLOCKED (2025-10-27) | Docs Guild, Ops Guild | DEPLOY-POLICY-27-002 | Write `/docs/runbooks/policy-incident.md` detailing rollback, freeze, forensic steps, notifications. | Runbook merged; rehearsal recorded; checklist appended. |
|
||||
> Blocked by `DEPLOY-POLICY-27-002` incident runbook inputs pending.
|
||||
> Blocker: Ops runbook inputs (`DEPLOY-POLICY-27-002`) pending.
|
||||
| DOCS-POLICY-27-013 | BLOCKED (2025-10-27) | Docs Guild, Policy Guild | CONSOLE-STUDIO-27-001, REGISTRY-API-27-002 | Update `/docs/examples/policy-templates.md` with new templates, snippets, and sample policies. | Examples committed with commentary; lint passes; checklist appended. |
|
||||
> Blocked by `CONSOLE-STUDIO-27-001`/`REGISTRY-API-27-002` templates missing.
|
||||
> Blocker: Studio templates and registry storage (`CONSOLE-STUDIO-27-001`, `REGISTRY-API-27-002`) not available.
|
||||
| DOCS-POLICY-27-014 | BLOCKED (2025-10-27) | Docs Guild, Policy Registry Guild | REGISTRY-API-27-003, WEB-POLICY-27-001 | Refresh `/docs/aoc/aoc-guardrails.md` to include Studio-specific guardrails and validation scenarios. | Doc updated with Studio guardrails; compliance checklist appended. |
|
||||
> Blocked by `REGISTRY-API-27-003` & `WEB-POLICY-27-001` guardrails not implemented.
|
||||
> Blocker: Registry compile pipeline/web proxy (`REGISTRY-API-27-003`, `WEB-POLICY-27-001`) outstanding.
|
||||
|
||||
## Vulnerability Explorer (Sprint 29)
|
||||
|
||||
| ID | Status | Owner(s) | Depends on | Description | Exit Criteria |
|
||||
|----|--------|----------|------------|-------------|---------------|
|
||||
| DOCS-VULN-29-001 | TODO | Docs Guild, Vuln Explorer Guild | VULN-API-29-001 | Publish `/docs/vuln/explorer-overview.md` covering domain model, identities, AOC guarantees, workflow summary. | Doc merged with diagrams/table; compliance checklist appended. |
|
||||
| DOCS-VULN-29-002 | TODO | Docs Guild, Console Guild | CONSOLE-VULN-29-001..006 | Write `/docs/vuln/explorer-using-console.md` with workflows, screenshots, keyboard shortcuts, saved views, deep links. | Doc merged; images stored; WCAG notes included; checklist appended. |
|
||||
| DOCS-VULN-29-003 | TODO | Docs Guild, Vuln Explorer API Guild | VULN-API-29-001..009 | Author `/docs/vuln/explorer-api.md` (endpoints, query schema, grouping, errors, rate limits). | Doc aligned with OpenAPI; examples validated; checklist appended. |
|
||||
| DOCS-VULN-29-004 | TODO | Docs Guild, DevEx/CLI Guild | CLI-VULN-29-001..005 | Publish `/docs/vuln/explorer-cli.md` with command reference, samples, exit codes, CI snippets. | CLI doc merged; transcripts/JSON outputs validated; checklist appended. |
|
||||
| DOCS-VULN-29-005 | TODO | Docs Guild, Findings Ledger Guild | LEDGER-29-001..009 | Write `/docs/vuln/findings-ledger.md` detailing event schema, hashing, Merkle roots, replay tooling. | Doc merged; compliance checklist appended; audit team sign-off. |
|
||||
| DOCS-VULN-29-006 | TODO | Docs Guild, Policy Guild | POLICY-ENGINE-29-001..003 | Update `/docs/policy/vuln-determinations.md` for new rationale, signals, simulation semantics. | Doc updated; examples validated; checklist appended. |
|
||||
| DOCS-VULN-29-007 | TODO | Docs Guild, Excititor Guild | EXCITITOR-VULN-29-001..004 | Publish `/docs/vex/explorer-integration.md` covering CSAF mapping, suppression precedence, status semantics. | Doc merged; compliance checklist appended. |
|
||||
| DOCS-VULN-29-008 | TODO | Docs Guild, Concelier Guild | CONCELIER-VULN-29-001..004 | Publish `/docs/advisories/explorer-integration.md` covering key normalization, withdrawn handling, provenance. | Doc merged; checklist appended. |
|
||||
| DOCS-VULN-29-009 | TODO | Docs Guild, SBOM Service Guild | SBOM-VULN-29-001..002 | Author `/docs/sbom/vuln-resolution.md` detailing version semantics, scope, paths, safe version hints. | Doc merged; ecosystem tables validated; checklist appended. |
|
||||
| DOCS-VULN-29-010 | TODO | Docs Guild, Observability Guild | VULN-API-29-009, DEVOPS-VULN-29-002 | Publish `/docs/observability/vuln-telemetry.md` (metrics, logs, tracing, dashboards, SLOs). | Doc merged; dashboards linked; checklist appended. |
|
||||
| DOCS-VULN-29-011 | TODO | Docs Guild, Security Guild | AUTH-VULN-29-001..003 | Create `/docs/security/vuln-rbac.md` for roles, ABAC policies, attachment encryption, CSRF. | Security doc approved; checklist appended. |
|
||||
| DOCS-VULN-29-012 | TODO | Docs Guild, Ops Guild | DEVOPS-VULN-29-002, SCHED-WORKER-29-003 | Write `/docs/runbooks/vuln-ops.md` (projector lag, resolver storms, export failures, policy activation). | Runbook merged; rehearsal recorded; checklist appended. |
|
||||
| DOCS-VULN-29-013 | TODO | Docs Guild, Deployment Guild | DEPLOY-VULN-29-001..002 | Update `/docs/install/containers.md` with Findings Ledger & Vuln Explorer API images, manifests, resource sizing, health checks. | Install doc updated; validation commands included; checklist appended. |
|
||||
|
||||
## VEX Lens (Sprint 30)
|
||||
| ID | Status | Owner(s) | Depends on | Description | Exit Criteria |
|
||||
|----|--------|----------|------------|-------------|---------------|
|
||||
| DOCS-VEX-30-001 | TODO | Docs Guild, VEX Lens Guild | VEXLENS-30-005 | Publish `/docs/vex/consensus-overview.md` describing purpose, scope, AOC guarantees. | Doc merged with diagrams/terminology tables; compliance checklist appended. |
|
||||
| DOCS-VEX-30-002 | TODO | Docs Guild, VEX Lens Guild | VEXLENS-30-005 | Author `/docs/vex/consensus-algorithm.md` covering normalization, weighting, thresholds, examples. | Doc merged; math reviewed by Policy; checklist appended. |
|
||||
| DOCS-VEX-30-003 | TODO | Docs Guild, Issuer Directory Guild | ISSUER-30-001..003 | Document `/docs/vex/issuer-directory.md` (issuer management, keys, trust overrides, audit). | Doc merged; security review done; checklist appended. |
|
||||
| DOCS-VEX-30-004 | TODO | Docs Guild, VEX Lens Guild | VEXLENS-30-007 | Publish `/docs/vex/consensus-api.md` with endpoint specs, query params, rate limits. | API doc aligned with OpenAPI; examples validated; checklist appended. |
|
||||
| DOCS-VEX-30-005 | TODO | Docs Guild, Console Guild | CONSOLE-VEX-30-001 | Write `/docs/vex/consensus-console.md` covering UI workflows, filters, conflicts, accessibility. | Doc merged; screenshots added; checklist appended. |
|
||||
| DOCS-VEX-30-006 | TODO | Docs Guild, Policy Guild | POLICY-ENGINE-29-001, VEXLENS-30-004 | Add `/docs/policy/vex-trust-model.md` detailing policy knobs, thresholds, simulation. | Doc merged; policy review completed; checklist appended. |
|
||||
| DOCS-VEX-30-007 | TODO | Docs Guild, SBOM Service Guild | VEXLENS-30-002 | Publish `/docs/sbom/vex-mapping.md` (CPE→purl strategy, edge cases, overrides). | Doc merged; mapping tables validated; checklist appended. |
|
||||
| DOCS-VEX-30-008 | TODO | Docs Guild, Security Guild | ISSUER-30-002, VEXLENS-30-003 | Deliver `/docs/security/vex-signatures.md` (verification flow, key rotation, audit). | Doc approved by Security; checklist appended. |
|
||||
| DOCS-VEX-30-009 | TODO | Docs Guild, DevOps Guild | VEXLENS-30-009, DEVOPS-VEX-30-001 | Create `/docs/runbooks/vex-ops.md` for recompute storms, mapping failures, signature errors. | Runbook merged; rehearsal logged; checklist appended. |
|
||||
|
||||
## Advisory AI (Sprint 31)
|
||||
|
||||
| ID | Status | Owner(s) | Depends on | Description | Exit Criteria |
|
||||
|----|--------|----------|------------|-------------|---------------|
|
||||
| DOCS-AIAI-31-001 | TODO | Docs Guild, Advisory AI Guild | AIAI-31-006 | Publish `/docs/advisory-ai/overview.md` covering capabilities, guardrails, RBAC. | Doc merged with diagrams; compliance checklist appended. |
|
||||
| DOCS-AIAI-31-002 | TODO | Docs Guild, Advisory AI Guild | AIAI-31-004 | Author `/docs/advisory-ai/architecture.md` detailing RAG pipeline, deterministics, caching, model options. | Doc merged; architecture review done; checklist appended. |
|
||||
| DOCS-AIAI-31-003 | TODO | Docs Guild, Advisory AI Guild | AIAI-31-006 | Write `/docs/advisory-ai/api.md` describing endpoints, schemas, errors, rate limits. | API doc aligned with OpenAPI; examples validated; checklist appended. |
|
||||
| DOCS-AIAI-31-004 | TODO | Docs Guild, Console Guild | CONSOLE-VULN-29-001, CONSOLE-VEX-30-001 | Create `/docs/advisory-ai/console.md` with screenshots, a11y notes, copy-as-ticket instructions. | Doc merged; images stored; checklist appended. |
|
||||
| DOCS-AIAI-31-005 | TODO | Docs Guild, DevEx/CLI Guild | CLI-VULN-29-001, CLI-VEX-30-001 | Publish `/docs/advisory-ai/cli.md` covering commands, exit codes, scripting patterns. | Doc merged; examples tested; checklist appended. |
|
||||
| DOCS-AIAI-31-006 | TODO | Docs Guild, Policy Guild | POLICY-ENGINE-31-001 | Update `/docs/policy/assistant-parameters.md` covering temperature, token limits, ranking weights, TTLs. | Doc merged; policy review done; checklist appended. |
|
||||
| DOCS-AIAI-31-007 | TODO | Docs Guild, Security Guild | AIAI-31-005 | Write `/docs/security/assistant-guardrails.md` detailing redaction, injection defense, logging. | Doc approved by Security; checklist appended. |
|
||||
| DOCS-AIAI-31-008 | TODO | Docs Guild, SBOM Service Guild | SBOM-AIAI-31-001 | Publish `/docs/sbom/remediation-heuristics.md` (feasibility scoring, blast radius). | Doc merged; heuristics reviewed; checklist appended. |
|
||||
| DOCS-AIAI-31-009 | TODO | Docs Guild, DevOps Guild | DEVOPS-AIAI-31-001 | Create `/docs/runbooks/assistant-ops.md` for warmup, cache priming, model outages, scaling. | Runbook merged; rehearsal logged; checklist appended. |
|
||||
|
||||
## Notifications Studio
|
||||
|
||||
| ID | Status | Owner(s) | Depends on | Description | Exit Criteria |
|
||||
|----|--------|----------|------------|-------------|---------------|
|
||||
| DOCS-NOTIFY-38-001 | DONE (2025-10-29) | Docs Guild, Notifications Service Guild | NOTIFY-SVC-38-001..004 | Publish `/docs/notifications/overview.md` and `/docs/notifications/architecture.md`, each ending with imposed rule reminder. | Docs merged; diagrams verified; imposed rule appended. |
|
||||
| DOCS-NOTIFY-39-002 | DONE (2025-10-29) | Docs Guild, Notifications Service Guild | NOTIFY-SVC-39-001..004 | Publish `/docs/notifications/rules.md`, `/docs/notifications/templates.md`, `/docs/notifications/digests.md` with examples and imposed rule line. | Docs merged; examples validated; imposed rule appended. |
|
||||
| DOCS-NOTIFY-40-001 | TODO | Docs Guild, Security Guild | AUTH-NOTIFY-38-001, NOTIFY-SVC-40-001..004 | Publish `/docs/notifications/channels.md`, `/docs/notifications/escalations.md`, `/docs/notifications/api.md`, `/docs/operations/notifier-runbook.md`, `/docs/security/notifications-hardening.md`; each ends with imposed rule line. | Docs merged; accessibility checks passed; imposed rule appended. |
|
||||
|
||||
## CLI Parity & Task Packs
|
||||
|
||||
| ID | Status | Owner(s) | Depends on | Description | Exit Criteria |
|
||||
|----|--------|----------|------------|-------------|---------------|
|
||||
| DOCS-CLI-41-001 | TODO | Docs Guild, DevEx/CLI Guild | CLI-CORE-41-001 | Publish `/docs/cli/overview.md`, `/docs/cli/configuration.md`, `/docs/cli/output-and-exit-codes.md` with imposed rule statements. | Docs merged; examples verified; imposed rule appended. |
|
||||
| DOCS-CLI-42-001 | TODO | Docs Guild | DOCS-CLI-41-001, CLI-PARITY-41-001 | Publish `/docs/cli/parity-matrix.md` and command guides under `/docs/cli/commands/*.md` (policy, sbom, vuln, vex, advisory, export, orchestrator, notify, aoc, auth). | Guides merged; parity automation documented; imposed rule appended. |
|
||||
| DOCS-PACKS-43-001 | DONE (2025-10-27) | Docs Guild, Task Runner Guild | PACKS-REG-42-001, TASKRUN-42-001 | Publish `/docs/task-packs/spec.md`, `/docs/task-packs/authoring-guide.md`, `/docs/task-packs/registry.md`, `/docs/task-packs/runbook.md`, `/docs/security/pack-signing-and-rbac.md`, `/docs/operations/cli-release-and-packaging.md` with imposed rule statements. | Docs merged; tutorials validated; imposed rule appended; cross-links added. |
|
||||
|
||||
## Containerized Distribution (Epic 13)
|
||||
|
||||
| ID | Status | Owner(s) | Depends on | Description | Exit Criteria |
|
||||
|----|--------|----------|------------|-------------|---------------|
|
||||
| DOCS-INSTALL-44-001 | TODO | Docs Guild, Deployment Guild | COMPOSE-44-001 | Publish `/docs/install/overview.md` and `/docs/install/compose-quickstart.md` with imposed rule line and copy-ready commands. | Docs merged; screenshots/commands verified; imposed rule appended. |
|
||||
| DOCS-INSTALL-45-001 | TODO | Docs Guild, Deployment Guild | HELM-45-001 | Publish `/docs/install/helm-prod.md` and `/docs/install/configuration-reference.md` with values tables and imposed rule reminder. | Docs merged; configuration matrix verified; imposed rule appended. |
|
||||
| DOCS-INSTALL-46-001 | TODO | Docs Guild, Security Guild | DEPLOY-PACKS-43-001, CLI-PACKS-43-001 | Publish `/docs/install/airgap.md`, `/docs/security/supply-chain.md`, `/docs/operations/health-and-readiness.md`, `/docs/release/image-catalog.md`, `/docs/console/onboarding.md` (each with imposed rule). | Docs merged; checksum/signature sections validated; imposed rule appended. |
|
||||
|
||||
## Authority-Backed Scopes & Tenancy (Epic 14)
|
||||
| ID | Status | Owner(s) | Depends on | Description | Exit Criteria |
|
||||
|----|--------|----------|------------|-------------|---------------|
|
||||
| DOCS-TEN-47-001 | TODO | Docs Guild, Authority Core | AUTH-TEN-47-001 | Publish `/docs/security/tenancy-overview.md` and `/docs/security/scopes-and-roles.md` outlining scope grammar, tenant model, imposed rule reminder. | Docs merged; diagrams included; imposed rule appended. |
|
||||
| DOCS-TEN-48-001 | TODO | Docs Guild, Platform Ops | WEB-TEN-48-001 | Publish `/docs/operations/multi-tenancy.md`, `/docs/operations/rls-and-data-isolation.md`, `/docs/console/admin-tenants.md`. | Docs merged; examples validated; imposed rule appended. |
|
||||
| DOCS-TEN-49-001 | TODO | Docs & DevEx Guilds | CLI-TEN-47-001, AUTH-TEN-49-001 | Publish `/docs/cli/authentication.md`, `/docs/api/authentication.md`, `/docs/policy/examples/abac-overlays.md`, update `/docs/install/configuration-reference.md` with new env vars, all ending with imposed rule line. | Docs merged; command examples verified; imposed rule appended. |
|
||||
# Docs Guild Task Board (UTC 2025-10-10)
|
||||
|
||||
| ID | Status | Owner(s) | Depends on | Description | Exit Criteria |
|
||||
|----|--------|----------|------------|-------------|---------------|
|
||||
| DOC7.README-INDEX | DONE (2025-10-17) | Docs Guild | — | Refresh index docs (docs/README.md + root README) after architecture dossier split and Offline Kit overhaul. | ✅ ToC reflects new component architecture docs; ✅ root README highlights updated doc set; ✅ Offline Kit guide linked correctly. |
|
||||
| DOC4.AUTH-PDG | DONE (2025-10-19) | Docs Guild, Plugin Team | PLG6.DOC | Copy-edit `docs/dev/31_AUTHORITY_PLUGIN_DEVELOPER_GUIDE.md`, export lifecycle diagram, add LDAP RFC cross-link. | ✅ PR merged with polish; ✅ Diagram committed; ✅ Slack handoff posted. |
|
||||
| DOC1.AUTH | DONE (2025-10-12) | Docs Guild, Authority Core | CORE5B.DOC | Draft `docs/11_AUTHORITY.md` covering architecture, configuration, bootstrap flows. | ✅ Architecture + config sections approved by Core; ✅ Samples reference latest options; ✅ Offline note added. |
|
||||
| DOC3.Concelier-Authority | DONE (2025-10-12) | Docs Guild, DevEx | FSR4 | Polish operator/runbook sections (DOC3/DOC5) to document Concelier authority rollout, bypass logging, and enforcement checklist. | ✅ DOC3/DOC5 updated with audit runbook references; ✅ enforcement deadline highlighted; ✅ Docs guild sign-off. |
|
||||
| DOC5.Concelier-Runbook | DONE (2025-10-12) | Docs Guild | DOC3.Concelier-Authority | Produce dedicated Concelier authority audit runbook covering log fields, monitoring recommendations, and troubleshooting steps. | ✅ Runbook published; ✅ linked from DOC3/DOC5; ✅ alerting guidance included. |
|
||||
| FEEDDOCS-DOCS-05-001 | DONE (2025-10-11) | Docs Guild | FEEDMERGE-ENGINE-04-001, FEEDMERGE-ENGINE-04-002 | Publish Concelier conflict resolution runbook covering precedence workflow, merge-event auditing, and Sprint 3 metrics. | ✅ `docs/ops/concelier-conflict-resolution.md` committed; ✅ metrics/log tables align with latest merge code; ✅ Ops alert guidance handed to Concelier team. |
|
||||
| FEEDDOCS-DOCS-05-002 | DONE (2025-10-16) | Docs Guild, Concelier Ops | FEEDDOCS-DOCS-05-001 | Ops sign-off captured: conflict runbook circulated, alert thresholds tuned, and rollout decisions documented in change log. | ✅ Ops review recorded; ✅ alert thresholds finalised using `docs/ops/concelier-authority-audit-runbook.md`; ✅ change-log entry linked from runbook once GHSA/NVD/OSV regression fixtures land. |
|
||||
| DOCS-ADR-09-001 | DONE (2025-10-19) | Docs Guild, DevEx | — | Establish ADR process (`docs/adr/0000-template.md`) and document usage guidelines. | Template published; README snippet linking ADR process; announcement posted (`docs/updates/2025-10-18-docs-guild.md`). |
|
||||
| DOCS-EVENTS-09-002 | DONE (2025-10-19) | Docs Guild, Platform Events | SCANNER-EVENTS-15-201 | Publish event schema catalog (`docs/events/`) for `scanner.report.ready@1`, `scheduler.rescan.delta@1`, `attestor.logged@1`. | Schemas validated (Ajv CI hooked); docs/events/README summarises usage; Platform Events notified via `docs/updates/2025-10-18-docs-guild.md`. |
|
||||
| DOCS-EVENTS-09-003 | DONE (2025-10-19) | Docs Guild | DOCS-EVENTS-09-002 | Add human-readable envelope field references and canonical payload samples for published events, including offline validation workflow. | Tables explain common headers/payload segments; versioned sample payloads committed; README links to validation instructions and samples. |
|
||||
| DOCS-EVENTS-09-004 | DONE (2025-10-19) | Docs Guild, Scanner WebService | SCANNER-EVENTS-15-201 | Refresh scanner event docs to mirror DSSE-backed report fields, document `scanner.scan.completed`, and capture canonical sample validation. | Schemas updated for new payload shape; README references DSSE reuse and validation test; samples align with emitted events. |
|
||||
| PLATFORM-EVENTS-09-401 | DONE (2025-10-21) | Platform Events Guild | DOCS-EVENTS-09-003 | Embed canonical event samples into contract/integration tests and ensure CI validates payloads against published schemas. | Notify models tests now run schema validation against `docs/events/*.json`, event schemas allow optional `attributes`, and docs capture the new validation workflow. |
|
||||
| RUNTIME-GUILD-09-402 | DONE (2025-10-19) | Runtime Guild | SCANNER-POLICY-09-107 | Confirm Scanner WebService surfaces `quietedFindingCount` and progress hints to runtime consumers; document readiness checklist. | Runtime verification run captures enriched payload; checklist/doc updates merged; stakeholders acknowledge availability. |
|
||||
| DOCS-CONCELIER-07-201 | DONE (2025-10-22) | Docs Guild, Concelier WebService | FEEDWEB-DOCS-01-001 | Final editorial review and publish pass for Concelier authority toggle documentation (Quickstart + operator guide). | Review feedback resolved, publish PR merged, release notes updated with documentation pointer. |
|
||||
| DOCS-RUNTIME-17-004 | DONE (2025-10-26) | Docs Guild, Runtime Guild | SCANNER-EMIT-17-701, ZASTAVA-OBS-17-005, DEVOPS-REL-17-002 | Document build-id workflows: SBOM exposure, runtime event payloads (`process.buildId`), Scanner `/policy/runtime` response (`buildIds` list), debug-store layout, and operator guidance for symbol retrieval. | Architecture + operator docs updated with build-id sections (Observer, Scanner, CLI), examples show `readelf` output + debuginfod usage, references linked from Offline Kit/Release guides + CLI help. |
|
||||
| DOCS-OBS-50-001 | BLOCKED (2025-10-26) | Docs Guild, Observability Guild | TELEMETRY-OBS-50-001 | Publish `/docs/observability/overview.md` introducing scope, imposed rule banner, architecture diagram, and tenant guarantees. | Doc merged with imposed rule banner; diagram committed; cross-links to telemetry stack + evidence locker docs. |
|
||||
> Blocked: waiting on telemetry core deliverable (TELEMETRY-OBS-50-001) to finalise architecture details and diagrams.
|
||||
| DOCS-OBS-50-002 | TODO | Docs Guild, Security Guild | TELEMETRY-OBS-50-002 | Author `/docs/observability/telemetry-standards.md` detailing common fields, scrubbing policy, sampling defaults, and redaction override procedure. | Doc merged; imposed rule banner present; examples validated with telemetry fixtures; security review sign-off captured. |
|
||||
| DOCS-OBS-50-003 | TODO | Docs Guild, Observability Guild | TELEMETRY-OBS-50-001 | Create `/docs/observability/logging.md` covering structured log schema, dos/don'ts, tenant isolation, and copyable examples. | Doc merged with banner; sample logs redacted; lint passes; linked from coding standards. |
|
||||
| DOCS-OBS-50-004 | TODO | Docs Guild, Observability Guild | TELEMETRY-OBS-50-002 | Draft `/docs/observability/tracing.md` explaining context propagation, async linking, CLI header usage, and sampling strategies. | Doc merged; imposed rule banner included; diagrams updated; references to CLI/Console features added. |
|
||||
| DOCS-OBS-51-001 | TODO | Docs Guild, DevOps Guild | WEB-OBS-51-001, DEVOPS-OBS-51-001 | Publish `/docs/observability/metrics-and-slos.md` cataloging metrics, SLO targets, burn rate policies, and alert runbooks. | Doc merged with banner; SLO tables verified; alert workflows linked to incident runbook. |
|
||||
| DOCS-SEC-OBS-50-001 | TODO | Docs Guild, Security Guild | TELEMETRY-OBS-51-002 | Update `/docs/security/redaction-and-privacy.md` to cover telemetry privacy controls, tenant opt-in debug, and imposed rule reminder. | Doc merged; redaction matrix updated; banner present; security sign-off recorded. |
|
||||
| DOCS-INSTALL-50-001 | TODO | Docs Guild, DevOps Guild | DEVOPS-OBS-50-003 | Add `/docs/install/telemetry-stack.md` with collector deployment, exporter options, offline kit notes, and imposed rule banner. | Doc merged; install steps verified on air-gapped profile; banner present; screenshots attached. |
|
||||
| DOCS-FORENSICS-53-001 | TODO | Docs Guild, Evidence Locker Guild | EVID-OBS-53-003 | Publish `/docs/forensics/evidence-locker.md` describing bundle formats, WORM options, retention, legal hold, and imposed rule banner. | Doc merged; manifest examples validated; banner present; legal hold steps aligned with API. |
|
||||
| DOCS-FORENSICS-53-002 | TODO | Docs Guild, Provenance Guild | PROV-OBS-54-001 | Release `/docs/forensics/provenance-attestation.md` covering DSSE schema, signing process, verification workflow, and imposed rule banner. | Doc merged; sample statements reference fixtures; banner included; verification steps tested. |
|
||||
| DOCS-FORENSICS-53-003 | TODO | Docs Guild, Timeline Indexer Guild | TIMELINE-OBS-52-003 | Publish `/docs/forensics/timeline.md` with schema, event kinds, filters, query examples, and imposed rule banner. | Doc merged; query examples validated; banner present; linked from Console/CLI docs. |
|
||||
| DOCS-CONSOLE-OBS-52-001 | TODO | Docs Guild, Console Guild | CONSOLE-OBS-51-001 | Document `/docs/console/observability.md` showcasing Observability Hub widgets, trace/log search, imposed rule banner, and accessibility tips. | Doc merged; screenshots updated; banner present; navigation steps verified. |
|
||||
| DOCS-CONSOLE-OBS-52-002 | TODO | Docs Guild, Console Guild | CONSOLE-OBS-52-002, CONSOLE-OBS-53-001 | Publish `/docs/console/forensics.md` covering timeline explorer, evidence viewer, attestation verifier, imposed rule banner, and troubleshooting. | Doc merged; banner included; workflows validated via Playwright capture; troubleshooting section populated. |
|
||||
| DOCS-CLI-OBS-52-001 | TODO | Docs Guild, DevEx/CLI Guild | CLI-OBS-52-001 | Create `/docs/cli/observability.md` detailing `stella obs` commands, examples, exit codes, imposed rule banner, and scripting tips. | Doc merged; examples tested; banner included; CLI parity matrix updated. |
|
||||
| DOCS-CLI-FORENSICS-53-001 | TODO | Docs Guild, DevEx/CLI Guild | CLI-FORENSICS-54-001 | Publish `/docs/cli/forensics.md` for snapshot/verify/attest commands with sample outputs, imposed rule banner, and offline workflows. | Doc merged; sample bundles verified; banner present; offline notes cross-linked. |
|
||||
| DOCS-RUNBOOK-55-001 | TODO | Docs Guild, Ops Guild | DEVOPS-OBS-55-001, WEB-OBS-55-001 | Author `/docs/runbooks/incidents.md` describing incident mode activation, escalation steps, retention impact, verification checklist, and imposed rule banner. | Doc merged; runbook rehearsed; banner included; linked from alerts. |
|
||||
| DOCS-AOC-19-001 | DONE (2025-10-26) | Docs Guild, Concelier Guild | CONCELIER-WEB-AOC-19-001, EXCITITOR-WEB-AOC-19-001 | Author `/docs/ingestion/aggregation-only-contract.md` covering philosophy, invariants, schemas, error codes, migration, observability, and security checklist. | New doc published with compliance checklist; cross-links from existing docs added. |
|
||||
| DOCS-AOC-19-002 | DONE (2025-10-26) | Docs Guild, Architecture Guild | DOCS-AOC-19-001 | Update `/docs/architecture/overview.md` to include AOC boundary, raw stores, and sequence diagram (fetch → guard → raw insert → policy evaluation). | Overview doc updated with diagrams/text; lint passes; stakeholders sign off. |
|
||||
| DOCS-AOC-19-003 | DONE (2025-10-26) | Docs Guild, Policy Guild | POLICY-AOC-19-003 | Refresh `/docs/architecture/policy-engine.md` clarifying ingestion boundary, raw inputs, and policy-only derived data. | Doc highlights raw-only ingestion contract, updated diagrams merge, compliance checklist added. |
|
||||
| DOCS-AOC-19-004 | DONE (2025-10-26) | Docs Guild, UI Guild | UI-AOC-19-001 | Extend `/docs/ui/console.md` with Sources dashboard tiles, violation drill-down workflow, and verification action. | UI doc updated with screenshots/flow descriptions, compliance checklist appended. |
|
||||
> DOCS-AOC-19-004: Architecture overview & policy-engine updates landed 2025-10-26; incorporate the new AOC boundary diagrams and metrics references.
|
||||
| DOCS-AOC-19-005 | DONE (2025-10-26) | Docs Guild, CLI Guild | CLI-AOC-19-003 | Update `/docs/cli/cli-reference.md` with `stella sources ingest --dry-run` and `stella aoc verify` usage, exit codes, and offline notes. | CLI reference + quickstart sections updated; examples validated; compliance checklist added. |
|
||||
> DOCS-AOC-19-005: New ingestion reference + architecture overview published 2025-10-26; ensure CLI docs link to both and surface AOC exit codes mapping.
|
||||
| DOCS-AOC-19-006 | DONE (2025-10-26) | Docs Guild, Observability Guild | CONCELIER-WEB-AOC-19-002, EXCITITOR-WEB-AOC-19-002 | Document new metrics/traces/log keys in `/docs/observability/observability.md`. | Observability doc lists new metrics/traces/log fields; dashboards referenced; compliance checklist appended. |
|
||||
| DOCS-AOC-19-007 | DONE (2025-10-26) | Docs Guild, Authority Core | AUTH-AOC-19-001 | Update `/docs/security/authority-scopes.md` with new ingestion scopes and tenancy enforcement notes. | Doc reflects new scopes, sample policies updated, compliance checklist added. |
|
||||
| DOCS-AOC-19-008 | DONE (2025-10-26) | Docs Guild, DevOps Guild | DEVOPS-AOC-19-002 | Refresh `/docs/deploy/containers.md` to cover validator enablement, guard env flags, and read-only verify user. | Deploy doc updated; offline kit section mentions validator scripts; compliance checklist appended. |
|
||||
| DOCS-AOC-19-009 | DONE (2025-10-26) | Docs Guild, Authority Core | AUTH-AOC-19-001 | Update AOC docs/samples to reflect new `advisory:*`, `vex:*`, and `aoc:verify` scopes. | Docs reference new scopes, samples aligned, compliance checklist updated. |
|
||||
|
||||
## Air-Gapped Mode (Epic 16)
|
||||
| ID | Status | Owner(s) | Depends on | Description | Exit Criteria |
|
||||
|----|--------|----------|------------|-------------|---------------|
|
||||
| DOCS-AIRGAP-56-001 | TODO | Docs Guild, AirGap Controller Guild | AIRGAP-CTL-56-002 | Publish `/docs/airgap/overview.md` outlining modes, lifecycle, responsibilities, and imposed rule banner. | Doc merged; banner present; diagrams included. |
|
||||
| DOCS-AIRGAP-56-002 | TODO | Docs Guild, DevOps Guild | DEVOPS-AIRGAP-56-001 | Author `/docs/airgap/sealing-and-egress.md` covering network policies, EgressPolicy facade usage, and verification steps. | Doc merged; examples validated; banner included. |
|
||||
| DOCS-AIRGAP-56-003 | TODO | Docs Guild, Exporter Guild | EXPORT-AIRGAP-56-001 | Create `/docs/airgap/mirror-bundles.md` describing bundle format, DSSE/TUF/Merkle validation, creation/import workflows. | Doc merged; sample commands verified; banner present. |
|
||||
| DOCS-AIRGAP-56-004 | TODO | Docs Guild, Deployment Guild | DEVOPS-AIRGAP-56-003 | Publish `/docs/airgap/bootstrap.md` detailing Bootstrap Pack creation, validation, and install procedures. | Doc merged; checklist appended; screenshots verified. |
|
||||
| DOCS-AIRGAP-57-001 | TODO | Docs Guild, AirGap Time Guild | AIRGAP-TIME-58-001 | Write `/docs/airgap/staleness-and-time.md` explaining time anchors, drift policies, staleness budgets, and UI indicators. | Doc merged; math checked; banner included. |
|
||||
| DOCS-AIRGAP-57-002 | TODO | Docs Guild, Console Guild | CONSOLE-AIRGAP-57-001 | Publish `/docs/console/airgap.md` covering sealed badge, import wizard, staleness dashboards. | Doc merged; screenshots captured; banner present. |
|
||||
| DOCS-AIRGAP-57-003 | TODO | Docs Guild, CLI Guild | CLI-AIRGAP-57-001 | Publish `/docs/cli/airgap.md` documenting commands, examples, exit codes. | Doc merged; examples validated; banner present. |
|
||||
| DOCS-AIRGAP-57-004 | TODO | Docs Guild, Ops Guild | DEVOPS-AIRGAP-56-002 | Create `/docs/airgap/operations.md` with runbooks for imports, failure recovery, and auditing. | Doc merged; runbooks rehearsed; banner included. |
|
||||
| DOCS-AIRGAP-58-001 | TODO | Docs Guild, Product Guild | CONSOLE-AIRGAP-58-002 | Provide `/docs/airgap/degradation-matrix.md` enumerating feature availability, fallbacks, remediation. | Doc merged; matrix reviewed; banner included. |
|
||||
| DOCS-AIRGAP-58-002 | TODO | Docs Guild, Security Guild | PROV-OBS-54-001 | Update `/docs/security/trust-and-signing.md` with DSSE/TUF roots, rotation, and signed time tokens. | Doc merged; security sign-off recorded; banner present. |
|
||||
| DOCS-AIRGAP-58-003 | TODO | Docs Guild, DevEx Guild | AIRGAP-POL-56-001 | Publish `/docs/dev/airgap-contracts.md` describing EgressPolicy usage, sealed-mode tests, linting. | Doc merged; sample code validated; banner included. |
|
||||
| DOCS-AIRGAP-58-004 | TODO | Docs Guild, Evidence Locker Guild | EVID-OBS-55-001 | Document `/docs/airgap/portable-evidence.md` for exporting/importing portable evidence bundles across enclaves. | Doc merged; verification steps tested; banner present. |
|
||||
|
||||
## SDKs & OpenAPI (Epic 17)
|
||||
| ID | Status | Owner(s) | Depends on | Description | Exit Criteria |
|
||||
|----|--------|----------|------------|-------------|---------------|
|
||||
| DOCS-OAS-61-001 | TODO | Docs Guild, API Contracts Guild | OAS-61-002 | Publish `/docs/api/overview.md` covering auth, tenancy, pagination, idempotency, rate limits with banner. | Doc merged; examples validated; banner present. |
|
||||
| DOCS-OAS-61-002 | TODO | Docs Guild, API Governance Guild | APIGOV-61-001 | Author `/docs/api/conventions.md` capturing naming, errors, filters, sorting, examples. | Doc merged; lint passes; banner included. |
|
||||
| DOCS-OAS-61-003 | TODO | Docs Guild, API Governance Guild | APIGOV-63-001 | Publish `/docs/api/versioning.md` describing SemVer, deprecation headers, migration playbooks. | Doc merged; example headers validated; banner present. |
|
||||
| DOCS-OAS-62-001 | TODO | Docs Guild, Developer Portal Guild | DEVPORT-62-002 | Stand up `/docs/api/reference/` auto-generated site; integrate with portal nav. | Reference site builds; search works; banner included. |
|
||||
| DOCS-SDK-62-001 | TODO | Docs Guild, SDK Generator Guild | SDKGEN-63-001 | Publish `/docs/sdks/overview.md` plus language guides (`typescript.md`, `python.md`, `go.md`, `java.md`). | Docs merged; code samples pulled from tested examples; banner present. |
|
||||
| DOCS-DEVPORT-62-001 | TODO | Docs Guild, Developer Portal Guild | DEVPORT-62-001 | Document `/docs/devportal/publishing.md` for build pipeline, offline bundle steps. | Doc merged; cross-links validated; banner included. |
|
||||
| DOCS-CONTRIB-62-001 | TODO | Docs Guild, API Governance Guild | APIGOV-61-001 | Publish `/docs/contributing/api-contracts.md` detailing how to edit OAS, lint rules, compatibility checks. | Doc merged; banner present; examples validated. |
|
||||
| DOCS-TEST-62-001 | TODO | Docs Guild, Contract Testing Guild | CONTR-62-001 | Author `/docs/testing/contract-testing.md` covering mock server, replay tests, golden fixtures. | Doc merged; references to tooling validated; banner present. |
|
||||
| DOCS-SEC-62-001 | TODO | Docs Guild, Authority Core | AUTH-AIRGAP-56-001 | Update `/docs/security/auth-scopes.md` with OAuth2/PAT scopes, tenancy header usage. | Doc merged; scope tables verified; banner included. |
|
||||
| DOCS-AIRGAP-DEVPORT-64-001 | TODO | Docs Guild, DevPortal Offline Guild | DVOFF-64-001 | Create `/docs/airgap/devportal-offline.md` describing offline bundle usage and verification. | Doc merged; verification steps tested; banner present. |
|
||||
|
||||
## Risk Profiles (Epic 18)
|
||||
| ID | Status | Owner(s) | Depends on | Description | Exit Criteria |
|
||||
|----|--------|----------|------------|-------------|---------------|
|
||||
| DOCS-RISK-66-001 | TODO | Docs Guild, Risk Profile Schema Guild | POLICY-RISK-66-001 | Publish `/docs/risk/overview.md` covering concepts and glossary. | Doc merged with banner; terminology reviewed. |
|
||||
| DOCS-RISK-66-002 | TODO | Docs Guild, Policy Guild | POLICY-RISK-66-003 | Author `/docs/risk/profiles.md` (authoring, versioning, scope). | Doc merged; schema examples validated; banner present. |
|
||||
| DOCS-RISK-66-003 | TODO | Docs Guild, Risk Engine Guild | RISK-ENGINE-67-001 | Publish `/docs/risk/factors.md` cataloging signals, transforms, reducers, TTLs. | Document merged; tables verified; banner included. |
|
||||
| DOCS-RISK-66-004 | TODO | Docs Guild, Risk Engine Guild | RISK-ENGINE-66-002 | Create `/docs/risk/formulas.md` detailing math, normalization, gating, severity. | Doc merged; equations rendered; banner present. |
|
||||
| DOCS-RISK-67-001 | TODO | Docs Guild, Risk Engine Guild | RISK-ENGINE-68-001 | Publish `/docs/risk/explainability.md` showing artifact schema and UI screenshots. | Doc merged; CLI examples validated; banner included. |
|
||||
| DOCS-RISK-67-002 | TODO | Docs Guild, API Guild | POLICY-RISK-67-002 | Produce `/docs/risk/api.md` with endpoint reference/examples. | Doc merged; OAS examples synced; banner present. |
|
||||
| DOCS-RISK-67-003 | TODO | Docs Guild, Console Guild | CONSOLE-RISK-66-001 | Document `/docs/console/risk-ui.md` for authoring, simulation, dashboards. | Doc merged; screenshots updated; banner included. |
|
||||
| DOCS-RISK-67-004 | TODO | Docs Guild, CLI Guild | CLI-RISK-66-001 | Publish `/docs/cli/risk.md` covering CLI workflows. | Doc merged; command examples validated; banner present. |
|
||||
| DOCS-RISK-68-001 | TODO | Docs Guild, Export Guild | RISK-BUNDLE-69-001 | Add `/docs/airgap/risk-bundles.md` for offline factor bundles. | Doc merged; verification steps confirmed; banner included. |
|
||||
| DOCS-RISK-68-002 | TODO | Docs Guild, Security Guild | POLICY-RISK-66-003 | Update `/docs/security/aoc-invariants.md` with risk scoring provenance guarantees. | Doc merged; audit references updated; banner present. |
|
||||
|
||||
## Attestor Console (Epic 19)
|
||||
| ID | Status | Owner(s) | Depends on | Description | Exit Criteria |
|
||||
|----|--------|----------|------------|-------------|---------------|
|
||||
| DOCS-ATTEST-73-001 | TODO | Docs Guild, Attestor Service Guild | ATTEST-TYPES-73-001 | Publish `/docs/attestor/overview.md` with imposed rule banner. | Doc merged; terminology validated. |
|
||||
| DOCS-ATTEST-73-002 | TODO | Docs Guild, Attestation Payloads Guild | ATTEST-TYPES-73-002 | Write `/docs/attestor/payloads.md` with schemas/examples. | Doc merged; examples validated via tests. |
|
||||
| DOCS-ATTEST-73-003 | TODO | Docs Guild, Policy Guild | POLICY-ATTEST-73-002 | Publish `/docs/attestor/policies.md` covering verification policies. | Doc merged; policy examples validated. |
|
||||
| DOCS-ATTEST-73-004 | TODO | Docs Guild, Attestor Service Guild | ATTESTOR-73-002 | Add `/docs/attestor/workflows.md` detailing ingest, verify, bulk operations. | Doc merged; workflows tested. |
|
||||
| DOCS-ATTEST-74-001 | TODO | Docs Guild, KMS Guild | KMS-73-001 | Publish `/docs/attestor/keys-and-issuers.md`. | Doc merged; rotation guidance verified. |
|
||||
| DOCS-ATTEST-74-002 | TODO | Docs Guild, Transparency Guild | TRANSP-74-001 | Document `/docs/attestor/transparency.md` with witness usage/offline validation. | Doc merged; proofs validated. |
|
||||
| DOCS-ATTEST-74-003 | TODO | Docs Guild, Attestor Console Guild | CONSOLE-ATTEST-73-001 | Write `/docs/console/attestor-ui.md` with screenshots/workflows. | Doc merged; screenshots captured; banner present. |
|
||||
| DOCS-ATTEST-74-004 | TODO | Docs Guild, CLI Attestor Guild | CLI-ATTEST-73-001 | Publish `/docs/cli/attest.md` covering CLI usage. | Doc merged; commands validated. |
|
||||
| DOCS-ATTEST-75-001 | TODO | Docs Guild, Export Attestation Guild | EXPORT-ATTEST-75-002 | Add `/docs/attestor/airgap.md` for attestation bundles. | Doc merged; verification steps confirmed. |
|
||||
| DOCS-ATTEST-75-002 | TODO | Docs Guild, Security Guild | ATTESTOR-73-002 | Update `/docs/security/aoc-invariants.md` with attestation invariants. | Doc merged; invariants detailed. |
|
||||
## Policy Engine v2
|
||||
|
||||
| ID | Status | Owner(s) | Depends on | Description | Exit Criteria |
|
||||
|----|--------|----------|------------|-------------|---------------|
|
||||
| DOCS-POLICY-20-001 | DONE (2025-10-26) | Docs Guild, Policy Guild | POLICY-ENGINE-20-000 | Author `/docs/policy/overview.md` covering concepts, inputs/outputs, determinism, and compliance checklist. | Doc published with diagrams + glossary; lint passes; checklist included. |
|
||||
| DOCS-POLICY-20-002 | DONE (2025-10-26) | Docs Guild, Policy Guild | POLICY-ENGINE-20-001 | Write `/docs/policy/dsl.md` with grammar, built-ins, examples, anti-patterns. | DSL doc includes grammar tables, examples, compliance checklist; validated against parser tests. |
|
||||
| DOCS-POLICY-20-003 | DONE (2025-10-26) | Docs Guild, Authority Core | AUTH-POLICY-20-001 | Publish `/docs/policy/lifecycle.md` describing draft→approve workflow, roles, audit, compliance list. | Lifecycle doc linked from UI/CLI help; approvals roles documented; checklist appended. |
|
||||
| DOCS-POLICY-20-004 | DONE (2025-10-26) | Docs Guild, Scheduler Guild | SCHED-MODELS-20-001 | Create `/docs/policy/runs.md` detailing run modes, incremental mechanics, cursors, replay. | Run doc includes sequence diagrams + compliance checklist; cross-links to scheduler docs. |
|
||||
| DOCS-POLICY-20-005 | DONE (2025-10-26) | Docs Guild, BE-Base Platform Guild | WEB-POLICY-20-001 | Draft `/docs/api/policy.md` describing endpoints, schemas, error codes. | API doc validated against OpenAPI; examples included; checklist appended. |
|
||||
| DOCS-POLICY-20-006 | DONE (2025-10-26) | Docs Guild, DevEx/CLI Guild | CLI-POLICY-20-002 | Produce `/docs/cli/policy.md` with command usage, exit codes, JSON output contracts. | CLI doc includes examples, exit codes, compliance checklist. |
|
||||
| DOCS-POLICY-20-007 | DONE (2025-10-26) | Docs Guild, UI Guild | UI-POLICY-20-001 | Document `/docs/ui/policy-editor.md` covering editor, simulation, diff workflows, approvals. | UI doc includes screenshots/placeholders, accessibility notes, compliance checklist. |
|
||||
| DOCS-POLICY-20-008 | DONE (2025-10-26) | Docs Guild, Architecture Guild | POLICY-ENGINE-20-003 | Write `/docs/architecture/policy-engine.md` (new epic content) with sequence diagrams, selection strategy, schema. | Architecture doc merged with diagrams; compliance checklist appended; references updated. |
|
||||
| DOCS-POLICY-20-009 | DONE (2025-10-26) | Docs Guild, Observability Guild | POLICY-ENGINE-20-007 | Add `/docs/observability/policy.md` for metrics/traces/logs, sample dashboards. | Observability doc includes metrics tables, dashboard screenshots, checklist. |
|
||||
| DOCS-POLICY-20-010 | DONE (2025-10-26) | Docs Guild, Security Guild | AUTH-POLICY-20-002 | Publish `/docs/security/policy-governance.md` covering scopes, approvals, tenancy, least privilege. | Security doc merged; compliance checklist appended; reviewed by Security Guild. |
|
||||
| DOCS-POLICY-20-011 | DONE (2025-10-26) | Docs Guild, Policy Guild | POLICY-ENGINE-20-001 | Populate `/docs/examples/policies/` with baseline/serverless/internal-only samples and commentary. | Example policies committed with explanations; lint passes; compliance checklist per file. |
|
||||
| DOCS-POLICY-20-012 | DONE (2025-10-26) | Docs Guild, Support Guild | WEB-POLICY-20-003 | Draft `/docs/faq/policy-faq.md` addressing common pitfalls, VEX conflicts, determinism issues. | FAQ published with Q/A entries, cross-links, compliance checklist. |
|
||||
|
||||
## Graph Explorer v1
|
||||
|
||||
| ID | Status | Owner(s) | Depends on | Description | Exit Criteria |
|
||||
|----|--------|----------|------------|-------------|---------------|
|
||||
|
||||
## Link-Not-Merge v1
|
||||
|
||||
| ID | Status | Owner(s) | Depends on | Description | Exit Criteria |
|
||||
|----|--------|----------|------------|-------------|---------------|
|
||||
| DOCS-LNM-22-001 | BLOCKED (2025-10-27) | Docs Guild, Concelier Guild | CONCELIER-LNM-21-001..003 | Author `/docs/advisories/aggregation.md` covering observation vs linkset, conflict handling, AOC requirements, and reviewer checklist. | Draft doc merged with examples + checklist; final sign-off blocked until Concelier schema/API tasks land. |
|
||||
> Blocker (2025-10-27): `CONCELIER-LNM-21-001..003` still TODO; update doc + fixtures once schema/API implementations are available.
|
||||
| DOCS-LNM-22-002 | BLOCKED (2025-10-27) | Docs Guild, Excititor Guild | EXCITITOR-LNM-21-001..003 | Publish `/docs/vex/aggregation.md` describing VEX observation/linkset model, product matching, conflicts. | Draft doc merged with fixtures; final approval blocked until Excititor observation/linkset work ships. |
|
||||
> Blocker (2025-10-27): `EXCITITOR-LNM-21-001..003` remain TODO; refresh doc, fixtures, and examples post-implementation.
|
||||
| DOCS-LNM-22-003 | BLOCKED (2025-10-27) | Docs Guild, BE-Base Platform Guild | WEB-LNM-21-001..003 | Update `/docs/api/advisories.md` and `/docs/api/vex.md` for new endpoints, parameters, errors, exports. | Draft pending gateway/API delivery; unblock once endpoints + OpenAPI specs are available. |
|
||||
> Blocker (2025-10-27): `WEB-LNM-21-001..003` all TODO—no gateway endpoints/OpenAPI to document yet.
|
||||
| DOCS-LNM-22-004 | TODO | Docs Guild, Policy Guild | POLICY-ENGINE-40-001 | Create `/docs/policy/effective-severity.md` detailing severity selection strategies from multiple sources. | Doc merged with policy examples; checklist included. |
|
||||
| DOCS-LNM-22-005 | BLOCKED (2025-10-27) | Docs Guild, UI Guild | UI-LNM-22-001..003 | Document `/docs/ui/evidence-panel.md` with screenshots, conflict badges, accessibility guidance. | Awaiting UI implementation to capture screenshots + flows; unblock once Evidence panel ships. |
|
||||
> Blocker (2025-10-27): `UI-LNM-22-001..003` all TODO; documentation requires final UI states and accessibility audit artifacts.
|
||||
|
||||
## StellaOps Console (Sprint 23)
|
||||
|
||||
| ID | Status | Owner(s) | Depends on | Description | Exit Criteria |
|
||||
|----|--------|----------|------------|-------------|---------------|
|
||||
| DOCS-CONSOLE-23-001 | DONE (2025-10-26) | Docs Guild, Console Guild | CONSOLE-CORE-23-004 | Publish `/docs/ui/console-overview.md` covering IA, tenant model, global filters, and AOC alignment with compliance checklist. | Doc merged with diagrams + overview tables; checklist appended; Console Guild sign-off. |
|
||||
| DOCS-CONSOLE-23-002 | DONE (2025-10-26) | Docs Guild, Console Guild | DOCS-CONSOLE-23-001 | Author `/docs/ui/navigation.md` detailing routes, breadcrumbs, keyboard shortcuts, deep links, and tenant context switching. | Navigation doc merged with shortcut tables and screenshots; accessibility checklist satisfied. |
|
||||
| DOCS-CONSOLE-23-003 | DONE (2025-10-26) | Docs Guild, SBOM Service Guild, Console Guild | SBOM-CONSOLE-23-001, CONSOLE-FEAT-23-102 | Document `/docs/ui/sbom-explorer.md` (catalog, detail, graph overlays, exports) including compliance checklist and performance tips. | Doc merged with annotated screenshots, export instructions, and overlay examples; checklist appended. |
|
||||
| DOCS-CONSOLE-23-004 | DONE (2025-10-26) | Docs Guild, Concelier Guild, Excititor Guild | CONCELIER-CONSOLE-23-001, EXCITITOR-CONSOLE-23-001 | Produce `/docs/ui/advisories-and-vex.md` explaining aggregation-not-merge, conflict indicators, raw viewers, and provenance banners. | Doc merged; raw JSON examples included; compliance checklist complete. |
|
||||
| DOCS-CONSOLE-23-005 | DONE (2025-10-26) | Docs Guild, Policy Guild | POLICY-CONSOLE-23-001, CONSOLE-FEAT-23-104 | Write `/docs/ui/findings.md` describing filters, saved views, explain drawer, exports, and CLI parity callouts. | Doc merged with filter matrix + explain walkthrough; checklist appended. |
|
||||
| DOCS-CONSOLE-23-006 | DONE (2025-10-26) | Docs Guild, Policy Guild, Product Ops | POLICY-CONSOLE-23-002, CONSOLE-FEAT-23-105 | Publish `/docs/ui/policies.md` with editor, simulation, approvals, compliance checklist, and RBAC mapping. | Doc merged; Monaco screenshots + simulation diff examples included; approval flow described; checklist appended. |
|
||||
| DOCS-CONSOLE-23-007 | DONE (2025-10-26) | Docs Guild, Scheduler Guild | SCHED-CONSOLE-23-001, CONSOLE-FEAT-23-106 | Document `/docs/ui/runs.md` covering queues, live progress, diffs, retries, evidence downloads, and troubleshooting. | Doc merged with SSE troubleshooting, metrics references, compliance checklist. |
|
||||
| DOCS-CONSOLE-23-008 | DONE (2025-10-26) | Docs Guild, Authority Guild | AUTH-CONSOLE-23-002, CONSOLE-FEAT-23-108 | Draft `/docs/ui/admin.md` describing users/roles, tenants, tokens, integrations, fresh-auth prompts, and RBAC mapping. | Doc merged with tables for scopes vs roles, screenshots, compliance checklist. |
|
||||
| DOCS-CONSOLE-23-009 | DONE (2025-10-27) | Docs Guild, DevOps Guild | DOWNLOADS-CONSOLE-23-001, CONSOLE-FEAT-23-109 | Publish `/docs/ui/downloads.md` listing product images, commands, offline instructions, parity with CLI, and compliance checklist. | Doc merged; manifest sample included; copy-to-clipboard guidance documented; checklist complete. |
|
||||
| DOCS-CONSOLE-23-010 | DONE (2025-10-27) | Docs Guild, Deployment Guild, Console Guild | DEVOPS-CONSOLE-23-002, CONSOLE-REL-23-301 | Write `/docs/deploy/console.md` (Helm, ingress, TLS, CSP, env vars, health checks) with compliance checklist. | Deploy doc merged; templates validated; CSP guidance included; checklist appended. |
|
||||
| DOCS-CONSOLE-23-011 | DONE (2025-10-28) | Docs Guild, Deployment Guild | DOCS-CONSOLE-23-010 | Update `/docs/install/docker.md` to cover Console image, Compose/Helm usage, offline tarballs, parity with CLI. | Doc updated with new sections; commands validated; compliance checklist appended. |
|
||||
| DOCS-CONSOLE-23-012 | DONE (2025-10-28) | Docs Guild, Security Guild | AUTH-CONSOLE-23-003, WEB-CONSOLE-23-002 | Publish `/docs/security/console-security.md` detailing OIDC flows, scopes, CSP, fresh-auth, evidence handling, and compliance checklist. | Security doc merged; threat model notes included; checklist appended. |
|
||||
| DOCS-CONSOLE-23-013 | DONE (2025-10-28) | Docs Guild, Observability Guild | TELEMETRY-CONSOLE-23-001, CONSOLE-QA-23-403 | Write `/docs/observability/ui-telemetry.md` cataloguing metrics/logs/traces, dashboards, alerts, and feature flags. | Doc merged with instrumentation tables, dashboard screenshots, checklist appended. |
|
||||
| DOCS-CONSOLE-23-014 | DONE (2025-10-28) | Docs Guild, Console Guild, CLI Guild | CONSOLE-DOC-23-502 | Maintain `/docs/cli-vs-ui-parity.md` matrix and integrate CI check guidance. | Matrix published with parity status, CI workflow documented, compliance checklist appended. |
|
||||
> 2025-10-28: Install Docker guide references pending CLI commands (`stella downloads manifest`, `stella downloads mirror`, `stella console status`). Update once CLI parity lands.
|
||||
| DOCS-CONSOLE-23-015 | DONE (2025-10-27) | Docs Guild, Architecture Guild | CONSOLE-CORE-23-001, WEB-CONSOLE-23-001 | Produce `/docs/architecture/console.md` describing frontend packages, data flow diagrams, SSE design, performance budgets. | Architecture doc merged with diagrams + compliance checklist; reviewers approve. |
|
||||
| DOCS-CONSOLE-23-016 | DONE (2025-10-28) | Docs Guild, Accessibility Guild | CONSOLE-QA-23-402, CONSOLE-FEAT-23-102 | Refresh `/docs/accessibility.md` with Console-specific keyboard flows, color tokens, testing tools, and compliance checklist updates. | Accessibility doc updated; audits referenced; checklist appended. |
|
||||
> 2025-10-28: Added guide covering keyboard matrix, screen reader behaviour, colour/focus tokens, testing workflow, offline guidance, and compliance checklist.
|
||||
| DOCS-CONSOLE-23-017 | DONE (2025-10-27) | Docs Guild, Console Guild | CONSOLE-FEAT-23-101..109 | Create `/docs/examples/ui-tours.md` providing triage, audit, policy rollout walkthroughs with annotated screenshots and GIFs. | UI tours doc merged; capture instructions + asset placeholders committed; compliance checklist appended. |
|
||||
| DOCS-CONSOLE-23-018 | DONE (2025-10-27) | Docs Guild, Security Guild | DOCS-CONSOLE-23-012 | Execute console security compliance checklist and capture Security Guild sign-off in Sprint 23 log. | Checklist completed; findings addressed or tickets filed; sign-off noted in updates file. |
|
||||
| DOCS-LNM-22-006 | DONE (2025-10-27) | Docs Guild, Architecture Guild | CONCELIER-LNM-21-001..005, EXCITITOR-LNM-21-001..005 | Refresh `/docs/architecture/conseiller.md` and `/docs/architecture/excitator.md` describing observation/linkset pipelines and event contracts. | Architecture docs updated with observation/linkset flow + event tables; revisit once service implementations land. |
|
||||
> Follow-up: align diagrams/examples after `CONCELIER-LNM-21` & `EXCITITOR-LNM-21` work merges (currently TODO).
|
||||
| DOCS-LNM-22-007 | TODO | Docs Guild, Observability Guild | CONCELIER-LNM-21-005, EXCITITOR-LNM-21-005, DEVOPS-LNM-22-002 | Publish `/docs/observability/aggregation.md` with metrics/traces/logs/SLOs. | Observability doc merged; dashboards referenced; checklist appended. |
|
||||
| DOCS-LNM-22-008 | TODO | Docs Guild, DevOps Guild | MERGE-LNM-21-001, CONCELIER-LNM-21-102 | Write `/docs/migration/no-merge.md` describing migration plan, backfill steps, rollback, feature flags. | Migration doc approved by stakeholders; checklist appended. |
|
||||
|
||||
## Policy Engine + Editor v1
|
||||
|
||||
| ID | Status | Owner(s) | Depends on | Description | Exit Criteria |
|
||||
|----|--------|----------|------------|-------------|---------------|
|
||||
| DOCS-POLICY-23-001 | TODO | Docs Guild, Policy Guild | POLICY-SPL-23-001..003 | Author `/docs/policy/overview.md` describing SPL philosophy, layering, and glossary with reviewer checklist. | Doc merged; lint passes; checklist appended. |
|
||||
| DOCS-POLICY-23-002 | TODO | Docs Guild, Policy Guild | POLICY-SPL-23-001 | Write `/docs/policy/spl-v1.md` (language reference, JSON Schema, examples). | Reference published with schema snippets; checklist completed. |
|
||||
| DOCS-POLICY-23-003 | TODO | Docs Guild, Policy Guild | POLICY-ENGINE-50-001..004 | Produce `/docs/policy/runtime.md` covering compiler, evaluator, caching, events, SLOs. | Runtime doc merged with diagrams; observability references included. |
|
||||
| DOCS-POLICY-23-004 | TODO | Docs Guild, UI Guild | UI-POLICY-23-001..006 | Document `/docs/policy/editor.md` (UI walkthrough, validation, simulation, approvals). | Editor doc merged with screenshots; accessibility checklist satisfied. |
|
||||
| DOCS-POLICY-23-005 | TODO | Docs Guild, Security Guild | AUTH-POLICY-23-001..002 | Publish `/docs/policy/governance.md` (roles, scopes, approvals, signing, exceptions). | Governance doc merged; checklist appended. |
|
||||
| DOCS-POLICY-23-006 | TODO | Docs Guild, BE-Base Platform Guild | WEB-POLICY-23-001..004 | Update `/docs/api/policy.md` with new endpoints, schemas, errors, pagination. | API doc aligns with OpenAPI; examples validated; checklist included. |
|
||||
| DOCS-POLICY-23-007 | TODO | Docs Guild, DevEx/CLI Guild | CLI-POLICY-23-004..006 | Update `/docs/cli/policy.md` for lint/simulate/activate/history commands, exit codes. | CLI doc updated; samples verified; checklist appended. |
|
||||
| DOCS-POLICY-23-008 | TODO | Docs Guild, Architecture Guild | POLICY-ENGINE-50-005..006 | Refresh `/docs/architecture/policy-engine.md` with data model, sequence diagrams, event flows. | Architecture doc merged with diagrams; checklist appended. |
|
||||
| DOCS-POLICY-23-009 | TODO | Docs Guild, DevOps Guild | MERGE-LNM-21-001, DEVOPS-LNM-22-001 | Create `/docs/migration/policy-parity.md` covering dual-run parity plan and rollback. | Migration doc approved; checklist appended. |
|
||||
| DOCS-POLICY-23-010 | TODO | Docs Guild, UI Guild | UI-POLICY-23-006 | Write `/docs/ui/explainers.md` showing explain trees, evidence overlays, interpretation guidance. | Doc merged with annotated screenshots; checklist appended. |
|
||||
|
||||
## Graph & Vuln Explorer v1
|
||||
|
||||
| ID | Status | Owner(s) | Depends on | Description | Exit Criteria |
|
||||
|----|--------|----------|------------|-------------|---------------|
|
||||
| DOCS-GRAPH-24-001 | TODO | Docs Guild, UI Guild | UI-GRAPH-24-001..006 | Author `/docs/ui/sbom-graph-explorer.md` detailing overlays, filters, saved views, accessibility, and AOC visibility. | Doc merged; screenshots included; checklist appended. |
|
||||
| DOCS-GRAPH-24-002 | TODO | Docs Guild, UI Guild | UI-GRAPH-24-005 | Publish `/docs/ui/vulnerability-explorer.md` covering table usage, grouping, fix suggestions, Why drawer. | Doc merged with annotated images; accessibility checklist satisfied. |
|
||||
| DOCS-GRAPH-24-003 | TODO | Docs Guild, SBOM Service Guild | SBOM-GRAPH-24-001..003 | Create `/docs/architecture/graph-index.md` describing data model, ingestion pipeline, caches, events. | Architecture doc merged with diagrams; checklist appended. |
|
||||
| DOCS-GRAPH-24-004 | TODO | Docs Guild, BE-Base Platform Guild | WEB-GRAPH-24-001..003 | Document `/docs/api/graph.md` and `/docs/api/vuln.md` avec endpoints, parameters, errors, RBAC. | API docs aligned with OpenAPI; examples validated; checklist appended. |
|
||||
| DOCS-GRAPH-24-005 | TODO | Docs Guild, DevEx/CLI Guild | CLI-GRAPH-24-001..003 | Update `/docs/cli/graph-and-vuln.md` covering new CLI commands, exit codes, scripting. | CLI doc merged; examples tested; checklist appended. |
|
||||
| DOCS-GRAPH-24-006 | TODO | Docs Guild, Policy Guild | POLICY-ENGINE-60-001..002 | Write `/docs/policy/ui-integration.md` explaining overlays, cache usage, simulator contracts. | Doc merged; references cross-linked; checklist appended. |
|
||||
| DOCS-GRAPH-24-007 | TODO | Docs Guild, DevOps Guild | DEVOPS-GRAPH-24-001..003 | Produce `/docs/migration/graph-parity.md` with rollout plan, parity checks, fallback guidance. | Migration doc approved; checklist appended. |
|
||||
|
||||
## Exceptions v1
|
||||
|
||||
| ID | Status | Owner(s) | Depends on | Description | Exit Criteria |
|
||||
|----|--------|----------|------------|-------------|---------------|
|
||||
| DOCS-EXC-25-001 | TODO | Docs Guild, Governance Guild | WEB-EXC-25-001 | Author `/docs/governance/exceptions.md` covering lifecycle, scope patterns, examples, compliance checklist. | Doc merged; reviewers sign off; checklist included. |
|
||||
| DOCS-EXC-25-002 | TODO | Docs Guild, Authority Core | AUTH-EXC-25-001 | Publish `/docs/governance/approvals-and-routing.md` detailing roles, routing matrix, MFA rules, audit trails. | Doc merged; routing examples validated; checklist appended. |
|
||||
| DOCS-EXC-25-003 | TODO | Docs Guild, BE-Base Platform Guild | WEB-EXC-25-001..003 | Create `/docs/api/exceptions.md` with endpoints, payloads, errors, idempotency notes. | API doc aligned with OpenAPI; examples tested; checklist appended. |
|
||||
| DOCS-EXC-25-004 | DONE (2025-10-27) | Docs Guild, Policy Guild | POLICY-ENGINE-70-001 | Document `/docs/policy/exception-effects.md` explaining evaluation order, conflicts, simulation. | Doc merged; tests cross-referenced; checklist appended. |
|
||||
| DOCS-EXC-25-005 | TODO | Docs Guild, UI Guild | UI-EXC-25-001..004 | Write `/docs/ui/exception-center.md` with UI walkthrough, badges, accessibility, shortcuts. | Doc merged with screenshots; accessibility checklist completed. |
|
||||
| DOCS-EXC-25-006 | TODO | Docs Guild, DevEx/CLI Guild | CLI-EXC-25-001..002 | Update `/docs/cli/exceptions.md` covering command usage and exit codes. | CLI doc updated; examples validated; checklist appended. |
|
||||
| DOCS-EXC-25-007 | TODO | Docs Guild, DevOps Guild | SCHED-WORKER-25-101, DEVOPS-GRAPH-24-003 | Publish `/docs/migration/exception-governance.md` describing cutover from legacy suppressions, notifications, rollback. | Migration doc approved; checklist included. |
|
||||
|
||||
> Update statuses (TODO/DOING/REVIEW/DONE/BLOCKED) as progress changes. Keep guides in sync with configuration samples under `etc/`.
|
||||
|
||||
> Remark (2025-10-13, DOC4.AUTH-PDG): Rate limit guide published (`docs/security/rate-limits.md`) and handed to plugin docs team for diagram uplift once PLG6.DIAGRAM lands.
|
||||
|
||||
## Orchestrator Dashboard (Epic 9)
|
||||
|
||||
| ID | Status | Owner(s) | Depends on | Description | Exit Criteria |
|
||||
|----|--------|----------|------------|-------------|---------------|
|
||||
| DOCS-ORCH-32-001 | TODO | Docs Guild | ORCH-SVC-32-001, AUTH-ORCH-32-001 | Author `/docs/orchestrator/overview.md` covering mission, roles, AOC alignment, governance, with imposed rule reminder. | Doc merged with diagrams; imposed rule statement included; entry linked from docs index. |
|
||||
| DOCS-ORCH-32-002 | TODO | Docs Guild | ORCH-SVC-32-002 | Author `/docs/orchestrator/architecture.md` detailing scheduler, DAGs, rate limits, data model, message bus, storage layout, restating imposed rule. | Architecture doc merged; diagrams reviewed; imposed rule noted. |
|
||||
| DOCS-ORCH-33-001 | TODO | Docs Guild | ORCH-SVC-33-001..004, WEB-ORCH-33-001 | Publish `/docs/orchestrator/api.md` (REST/WebSocket endpoints, payloads, error codes) with imposed rule note. | API doc merged; examples validated; imposed rule appended. |
|
||||
| DOCS-ORCH-33-002 | TODO | Docs Guild | CONSOLE-ORCH-32-002, CONSOLE-ORCH-33-001..002 | Publish `/docs/orchestrator/console.md` covering screens, a11y, live updates, control actions, reiterating imposed rule. | Console doc merged with screenshots; accessibility checklist done; imposed rule statement present. |
|
||||
| DOCS-ORCH-33-003 | TODO | Docs Guild | CLI-ORCH-33-001 | Publish `/docs/orchestrator/cli.md` documenting commands, options, exit codes, streaming output, offline usage, and imposed rule. | CLI doc merged; examples tested; imposed rule appended. |
|
||||
| DOCS-ORCH-34-001 | TODO | Docs Guild | ORCH-SVC-34-002, LEDGER-34-101 | Author `/docs/orchestrator/run-ledger.md` covering ledger schema, provenance chain, audit workflows, with imposed rule reminder. | Run-ledger doc merged; payload samples validated; imposed rule included; cross-links added. |
|
||||
| DOCS-ORCH-34-002 | TODO | Docs Guild | AUTH-ORCH-32-001, AUTH-ORCH-34-001 | Update `/docs/security/secrets-handling.md` for orchestrator KMS refs, redaction badges, operator hygiene, reiterating imposed rule. | Security doc merged; checklists updated; imposed rule restated; references from Console/CLI docs added. |
|
||||
| DOCS-ORCH-34-003 | TODO | Docs Guild | ORCH-SVC-33-003, ORCH-SVC-34-001, DEVOPS-ORCH-34-001 | Publish `/docs/operations/orchestrator-runbook.md` (incident playbook, backfill guide, circuit breakers, throttling) with imposed rule statement. | Runbook merged; steps validated with DevOps; imposed rule included; runbook linked from ops index. |
|
||||
| DOCS-ORCH-34-004 | TODO | Docs Guild | ORCH-SVC-32-005, WORKER-GO-33-001, WORKER-PY-33-001 | Document `/docs/schemas/artifacts.md` describing artifact kinds, schema versions, hashing, storage layout, restating imposed rule. | Schema doc merged; JSON schema provided; imposed rule included; sample payload validated. |
|
||||
| DOCS-ORCH-34-005 | TODO | Docs Guild | ORCH-SVC-34-001, DEVOPS-ORCH-34-001 | Author `/docs/slo/orchestrator-slo.md` defining SLOs, burn alerts, measurement, and reiterating imposed rule. | SLO doc merged; dashboard screenshots embedded; imposed rule appended; alerts documented. |
|
||||
|
||||
## Export Center (Epic 10)
|
||||
|
||||
| ID | Status | Owner(s) | Depends on | Description | Exit Criteria |
|
||||
|----|--------|----------|------------|-------------|---------------|
|
||||
| DOCS-EXPORT-35-001 | DONE (2025-10-29) | Docs Guild | EXPORT-SVC-35-001..006 | Author `/docs/export-center/overview.md` covering purpose, profiles, security, AOC alignment, surfaces, ending with imposed rule statement. | Doc merged with diagrams/examples; imposed rule line present; index updated. |
|
||||
| DOCS-EXPORT-35-002 | DONE (2025-10-29) | Docs Guild | EXPORT-SVC-35-002..005 | Publish `/docs/export-center/architecture.md` describing planner, adapters, manifests, signing, distribution flows, restating imposed rule. | Architecture doc merged; sequence diagrams included; rule statement appended. |
|
||||
| DOCS-EXPORT-35-003 | DONE (2025-10-29) | Docs Guild | EXPORT-SVC-35-003..004 | Publish `/docs/export-center/profiles.md` detailing schema fields, examples, compatibility, and imposed rule reminder. | Profiles doc merged; JSON schemas linked; imposed rule noted. |
|
||||
| DOCS-EXPORT-36-004 | DONE (2025-10-29) | Docs Guild | EXPORT-SVC-36-001..004, WEB-EXPORT-36-001 | Publish `/docs/export-center/api.md` covering endpoints, payloads, errors, and mention imposed rule. | API doc merged; examples validated; rule included. |
|
||||
| DOCS-EXPORT-36-005 | DONE (2025-10-29) | Docs Guild | CLI-EXPORT-35-001, CLI-EXPORT-36-001 | Publish `/docs/export-center/cli.md` with command reference, CI scripts, verification steps, restating imposed rule. | CLI doc merged; script snippets tested; rule appended. |
|
||||
| DOCS-EXPORT-36-006 | DONE (2025-10-29) | Docs Guild | EXPORT-SVC-36-001, DEVOPS-EXPORT-36-001 | Publish `/docs/export-center/trivy-adapter.md` covering field mappings, compatibility matrix, and imposed rule reminder. | Doc merged; mapping tables validated; rule included. |
|
||||
| DOCS-EXPORT-37-001 | DONE (2025-10-29) | Docs Guild | EXPORT-SVC-37-001, DEVOPS-EXPORT-37-001 | Publish `/docs/export-center/mirror-bundles.md` describing filesystem/OCI layouts, delta/encryption, import guide, ending with imposed rule. | Doc merged; diagrams provided; verification steps tested; rule stated. |
|
||||
| DOCS-EXPORT-37-002 | DONE (2025-10-29) | Docs Guild | EXPORT-SVC-35-005, EXPORT-SVC-37-002 | Publish `/docs/export-center/provenance-and-signing.md` detailing manifests, attestation flow, verification, reiterating imposed rule. | Doc merged; signature examples validated; rule appended. |
|
||||
| DOCS-EXPORT-37-003 | DONE (2025-10-29) | Docs Guild | DEVOPS-EXPORT-37-001 | Publish `/docs/operations/export-runbook.md` covering failures, tuning, capacity planning, with imposed rule reminder. | Runbook merged; procedures validated; rule included. |
|
||||
| DOCS-EXPORT-37-004 | TODO | Docs Guild | AUTH-EXPORT-37-001, EXPORT-SVC-37-002 | Publish `/docs/security/export-hardening.md` outlining RBAC, tenancy, encryption, redaction, restating imposed rule. | Security doc merged; checklist updated; rule appended. |
|
||||
| DOCS-EXPORT-37-101 | TODO | Docs Guild, DevEx/CLI Guild | CLI-EXPORT-37-001 | Refresh CLI verification sections once `stella export verify` lands (flags, exit codes, samples). | `docs/export-center/cli.md` & `docs/export-center/provenance-and-signing.md` updated with final command syntax; examples tested; rule reminder retained. |
|
||||
| DOCS-EXPORT-37-102 | TODO | Docs Guild, DevOps Guild | DEVOPS-EXPORT-37-001 | Embed export dashboards/alerts references into provenance/runbook docs after Grafana work ships. | Docs updated with dashboard IDs/alert notes; update logged; rule reminder present. |
|
||||
| DOCS-EXPORT-37-005 | TODO | Docs Guild, Exporter Service Guild | EXPORT-SVC-35-006, DEVOPS-EXPORT-36-001 | Validate Export Center docs against live Trivy/mirror bundles once implementation lands; refresh examples and CLI snippets accordingly. | Real bundle examples recorded; docs updated; verification steps confirmed with production artefacts. |
|
||||
> Note (2025-10-29): Blocked until exporter API (`EXPORT-SVC-35-006`) and Trivy/mirror adapters (`EXPORT-SVC-36-001`, `EXPORT-SVC-37-001`) ship. Requires access to CI smoke outputs (`DEVOPS-EXPORT-36-001`) for verification artifacts.
|
||||
|
||||
## Reachability v1
|
||||
|
||||
| ID | Status | Owner(s) | Depends on | Description | Exit Criteria |
|
||||
|----|--------|----------|------------|-------------|---------------|
|
||||
| DOCS-SIG-26-001 | TODO | Docs Guild, Signals Guild | SIGNALS-24-004 | Write `/docs/signals/reachability.md` covering states, scores, provenance, retention. | Doc merged with diagrams/examples; checklist appended. |
|
||||
| DOCS-SIG-26-002 | TODO | Docs Guild, Signals Guild | SIGNALS-24-002 | Publish `/docs/signals/callgraph-formats.md` with schemas and validation errors. | Doc merged; examples tested; checklist included. |
|
||||
| DOCS-SIG-26-003 | TODO | Docs Guild, Runtime Guild | SIGNALS-24-003 | Create `/docs/signals/runtime-facts.md` detailing agent capabilities, privacy safeguards, opt-in flags. | Doc merged; privacy review done; checklist appended. |
|
||||
| DOCS-SIG-26-004 | TODO | Docs Guild, Policy Guild | POLICY-ENGINE-80-001 | Document `/docs/policy/signals-weighting.md` for SPL predicates and weighting strategies. | Doc merged; sample policies validated; checklist appended. |
|
||||
| DOCS-SIG-26-005 | TODO | Docs Guild, UI Guild | UI-SIG-26-001..003 | Draft `/docs/ui/reachability-overlays.md` with badges, timelines, shortcuts. | Doc merged with screenshots; accessibility checklist completed. |
|
||||
| DOCS-SIG-26-006 | TODO | Docs Guild, DevEx/CLI Guild | CLI-SIG-26-001..002 | Update `/docs/cli/reachability.md` for new commands and automation recipes. | Doc merged; examples verified; checklist appended. |
|
||||
| DOCS-SIG-26-007 | TODO | Docs Guild, BE-Base Platform Guild | WEB-SIG-26-001..003 | Publish `/docs/api/signals.md` covering endpoints, payloads, ETags, errors. | API doc aligned with OpenAPI; examples tested; checklist appended. |
|
||||
| DOCS-SIG-26-008 | TODO | Docs Guild, DevOps Guild | DEVOPS-SIG-26-001..002 | Write `/docs/migration/enable-reachability.md` guiding rollout, fallbacks, monitoring. | Migration doc approved; checklist appended. |
|
||||
|
||||
## Policy Studio (Sprint 27)
|
||||
|
||||
| ID | Status | Owner(s) | Depends on | Description | Exit Criteria |
|
||||
|----|--------|----------|------------|-------------|---------------|
|
||||
| DOCS-POLICY-27-001 | BLOCKED (2025-10-27) | Docs Guild, Policy Guild | REGISTRY-API-27-001, POLICY-ENGINE-27-001 | Publish `/docs/policy/studio-overview.md` covering lifecycle, roles, glossary, and compliance checklist. | Doc merged with diagrams + lifecycle table; checklist appended; stakeholders sign off. |
|
||||
> Blocked by `REGISTRY-API-27-001` and `POLICY-ENGINE-27-001`; need spec + compile data.
|
||||
> Blocker: Registry OpenAPI (`REGISTRY-API-27-001`) and policy compile enrichments (`POLICY-ENGINE-27-001`) are still TODO; need final interfaces before drafting overview.
|
||||
| DOCS-POLICY-27-002 | BLOCKED (2025-10-27) | Docs Guild, Console Guild | CONSOLE-STUDIO-27-001 | Write `/docs/policy/authoring.md` detailing workspace templates, snippets, lint rules, IDE shortcuts, and best practices. | Authoring doc includes annotated screenshots, snippet catalog, compliance checklist. |
|
||||
> Blocked by `CONSOLE-STUDIO-27-001` Studio authoring UI pending.
|
||||
> Blocker: Console Studio authoring UI (`CONSOLE-STUDIO-27-001`) not implemented; awaiting UX to capture flows/snippets.
|
||||
| DOCS-POLICY-27-003 | BLOCKED (2025-10-27) | Docs Guild, Policy Registry Guild | REGISTRY-API-27-007 | Document `/docs/policy/versioning-and-publishing.md` (semver rules, attestations, rollback) with compliance checklist. | Doc merged with flow diagrams; attestation steps documented; checklist appended. |
|
||||
> Blocked by `REGISTRY-API-27-007` publish/sign pipeline outstanding.
|
||||
> Blocker: Registry publish/sign workflow (`REGISTRY-API-27-007`) pending.
|
||||
| DOCS-POLICY-27-004 | BLOCKED (2025-10-27) | Docs Guild, Scheduler Guild | REGISTRY-API-27-005, SCHED-WORKER-27-301 | Write `/docs/policy/simulation.md` covering quick vs batch sim, thresholds, evidence bundles, CLI examples. | Simulation doc includes charts, sample manifests, checklist appended. |
|
||||
> Blocked by `REGISTRY-API-27-005`/`SCHED-WORKER-27-301` batch simulation not ready.
|
||||
> Blocker: Batch simulation APIs/workers (`REGISTRY-API-27-005`, `SCHED-WORKER-27-301`) still TODO.
|
||||
| DOCS-POLICY-27-005 | BLOCKED (2025-10-27) | Docs Guild, Product Ops | REGISTRY-API-27-006 | Publish `/docs/policy/review-and-approval.md` with approver requirements, comments, webhooks, audit trail guidance. | Doc merged with role matrix + webhook schema; checklist appended. |
|
||||
> Blocked by `REGISTRY-API-27-006` review workflow not implemented.
|
||||
> Blocker: Review workflow (`REGISTRY-API-27-006`) not landed.
|
||||
| DOCS-POLICY-27-006 | BLOCKED (2025-10-27) | Docs Guild, Policy Guild | REGISTRY-API-27-008 | Author `/docs/policy/promotion.md` covering environments, canary, rollback, and monitoring steps. | Promotion doc includes examples + checklist; verified by Policy Ops. |
|
||||
> Blocked by `REGISTRY-API-27-008` promotion APIs pending.
|
||||
> Blocker: Promotion/canary APIs (`REGISTRY-API-27-008`) outstanding.
|
||||
| DOCS-POLICY-27-007 | BLOCKED (2025-10-27) | Docs Guild, DevEx/CLI Guild | CLI-POLICY-27-001..004 | Update `/docs/policy/cli.md` with new commands, JSON schemas, CI usage, and compliance checklist. | CLI doc merged with transcripts; schema references validated; checklist appended. |
|
||||
> Blocked by `CLI-POLICY-27-001..004` CLI commands missing.
|
||||
> Blocker: Policy CLI commands (`CLI-POLICY-27-001..004`) yet to implement.
|
||||
| DOCS-POLICY-27-008 | BLOCKED (2025-10-27) | Docs Guild, Policy Registry Guild | REGISTRY-API-27-001..008 | Publish `/docs/policy/api.md` describing Registry endpoints, request/response schemas, errors, and feature flags. | API doc aligned with OpenAPI; examples validated; checklist appended. |
|
||||
> Blocked by `REGISTRY-API-27-001..008` OpenAPI + endpoints incomplete.
|
||||
> Blocker: Registry OpenAPI/spec suite (`REGISTRY-API-27-001..008`) incomplete.
|
||||
| DOCS-POLICY-27-009 | BLOCKED (2025-10-27) | Docs Guild, Security Guild | AUTH-POLICY-27-002 | Create `/docs/security/policy-attestations.md` covering signing, verification, key rotation, and compliance checklist. | Security doc approved by Security Guild; verifier steps documented; checklist appended. |
|
||||
> Blocked by `AUTH-POLICY-27-002` signing enforcement pending.
|
||||
> Blocker: Authority signing enforcement (`AUTH-POLICY-27-002`) pending.
|
||||
| DOCS-POLICY-27-010 | BLOCKED (2025-10-27) | Docs Guild, Architecture Guild | REGISTRY-API-27-001, SCHED-WORKER-27-301 | Author `/docs/architecture/policy-registry.md` (service design, schemas, queues, failure modes) with diagrams and checklist. | Architecture doc merged; diagrams committed; checklist appended. |
|
||||
> Blocked by `REGISTRY-API-27-001` & `SCHED-WORKER-27-301` need delivery.
|
||||
> Blocker: Policy Registry schema/workers not delivered (see `REGISTRY-API-27-001`, `SCHED-WORKER-27-301`).
|
||||
| DOCS-POLICY-27-011 | BLOCKED (2025-10-27) | Docs Guild, Observability Guild | DEVOPS-POLICY-27-004 | Publish `/docs/observability/policy-telemetry.md` with metrics/log tables, dashboards, alerts, and compliance checklist. | Observability doc merged; dashboards linked; checklist appended. |
|
||||
> Blocked by `DEVOPS-POLICY-27-004` observability dashboards outstanding.
|
||||
> Blocker: Observability dashboards (`DEVOPS-POLICY-27-004`) not built.
|
||||
| DOCS-POLICY-27-012 | BLOCKED (2025-10-27) | Docs Guild, Ops Guild | DEPLOY-POLICY-27-002 | Write `/docs/runbooks/policy-incident.md` detailing rollback, freeze, forensic steps, notifications. | Runbook merged; rehearsal recorded; checklist appended. |
|
||||
> Blocked by `DEPLOY-POLICY-27-002` incident runbook inputs pending.
|
||||
> Blocker: Ops runbook inputs (`DEPLOY-POLICY-27-002`) pending.
|
||||
| DOCS-POLICY-27-013 | BLOCKED (2025-10-27) | Docs Guild, Policy Guild | CONSOLE-STUDIO-27-001, REGISTRY-API-27-002 | Update `/docs/examples/policy-templates.md` with new templates, snippets, and sample policies. | Examples committed with commentary; lint passes; checklist appended. |
|
||||
> Blocked by `CONSOLE-STUDIO-27-001`/`REGISTRY-API-27-002` templates missing.
|
||||
> Blocker: Studio templates and registry storage (`CONSOLE-STUDIO-27-001`, `REGISTRY-API-27-002`) not available.
|
||||
| DOCS-POLICY-27-014 | BLOCKED (2025-10-27) | Docs Guild, Policy Registry Guild | REGISTRY-API-27-003, WEB-POLICY-27-001 | Refresh `/docs/aoc/aoc-guardrails.md` to include Studio-specific guardrails and validation scenarios. | Doc updated with Studio guardrails; compliance checklist appended. |
|
||||
> Blocked by `REGISTRY-API-27-003` & `WEB-POLICY-27-001` guardrails not implemented.
|
||||
> Blocker: Registry compile pipeline/web proxy (`REGISTRY-API-27-003`, `WEB-POLICY-27-001`) outstanding.
|
||||
|
||||
## Vulnerability Explorer (Sprint 29)
|
||||
|
||||
| ID | Status | Owner(s) | Depends on | Description | Exit Criteria |
|
||||
|----|--------|----------|------------|-------------|---------------|
|
||||
| DOCS-VULN-29-001 | TODO | Docs Guild, Vuln Explorer Guild | VULN-API-29-001 | Publish `/docs/vuln/explorer-overview.md` covering domain model, identities, AOC guarantees, workflow summary. | Doc merged with diagrams/table; compliance checklist appended. |
|
||||
| DOCS-VULN-29-002 | TODO | Docs Guild, Console Guild | CONSOLE-VULN-29-001..006 | Write `/docs/vuln/explorer-using-console.md` with workflows, screenshots, keyboard shortcuts, saved views, deep links. | Doc merged; images stored; WCAG notes included; checklist appended. |
|
||||
| DOCS-VULN-29-003 | TODO | Docs Guild, Vuln Explorer API Guild | VULN-API-29-001..009 | Author `/docs/vuln/explorer-api.md` (endpoints, query schema, grouping, errors, rate limits). | Doc aligned with OpenAPI; examples validated; checklist appended. |
|
||||
| DOCS-VULN-29-004 | TODO | Docs Guild, DevEx/CLI Guild | CLI-VULN-29-001..005 | Publish `/docs/vuln/explorer-cli.md` with command reference, samples, exit codes, CI snippets. | CLI doc merged; transcripts/JSON outputs validated; checklist appended. |
|
||||
| DOCS-VULN-29-005 | TODO | Docs Guild, Findings Ledger Guild | LEDGER-29-001..009 | Write `/docs/vuln/findings-ledger.md` detailing event schema, hashing, Merkle roots, replay tooling. | Doc merged; compliance checklist appended; audit team sign-off. |
|
||||
| DOCS-VULN-29-006 | TODO | Docs Guild, Policy Guild | POLICY-ENGINE-29-001..003 | Update `/docs/policy/vuln-determinations.md` for new rationale, signals, simulation semantics. | Doc updated; examples validated; checklist appended. |
|
||||
| DOCS-VULN-29-007 | TODO | Docs Guild, Excititor Guild | EXCITITOR-VULN-29-001..004 | Publish `/docs/vex/explorer-integration.md` covering CSAF mapping, suppression precedence, status semantics. | Doc merged; compliance checklist appended. |
|
||||
| DOCS-VULN-29-008 | TODO | Docs Guild, Concelier Guild | CONCELIER-VULN-29-001..004 | Publish `/docs/advisories/explorer-integration.md` covering key normalization, withdrawn handling, provenance. | Doc merged; checklist appended. |
|
||||
| DOCS-VULN-29-009 | TODO | Docs Guild, SBOM Service Guild | SBOM-VULN-29-001..002 | Author `/docs/sbom/vuln-resolution.md` detailing version semantics, scope, paths, safe version hints. | Doc merged; ecosystem tables validated; checklist appended. |
|
||||
| DOCS-VULN-29-010 | TODO | Docs Guild, Observability Guild | VULN-API-29-009, DEVOPS-VULN-29-002 | Publish `/docs/observability/vuln-telemetry.md` (metrics, logs, tracing, dashboards, SLOs). | Doc merged; dashboards linked; checklist appended. |
|
||||
| DOCS-VULN-29-011 | TODO | Docs Guild, Security Guild | AUTH-VULN-29-001..003 | Create `/docs/security/vuln-rbac.md` for roles, ABAC policies, attachment encryption, CSRF. | Security doc approved; checklist appended. |
|
||||
| DOCS-VULN-29-012 | TODO | Docs Guild, Ops Guild | DEVOPS-VULN-29-002, SCHED-WORKER-29-003 | Write `/docs/runbooks/vuln-ops.md` (projector lag, resolver storms, export failures, policy activation). | Runbook merged; rehearsal recorded; checklist appended. |
|
||||
| DOCS-VULN-29-013 | TODO | Docs Guild, Deployment Guild | DEPLOY-VULN-29-001..002 | Update `/docs/install/containers.md` with Findings Ledger & Vuln Explorer API images, manifests, resource sizing, health checks. | Install doc updated; validation commands included; checklist appended. |
|
||||
|
||||
## VEX Lens (Sprint 30)
|
||||
| ID | Status | Owner(s) | Depends on | Description | Exit Criteria |
|
||||
|----|--------|----------|------------|-------------|---------------|
|
||||
| DOCS-VEX-30-001 | TODO | Docs Guild, VEX Lens Guild | VEXLENS-30-005 | Publish `/docs/vex/consensus-overview.md` describing purpose, scope, AOC guarantees. | Doc merged with diagrams/terminology tables; compliance checklist appended. |
|
||||
| DOCS-VEX-30-002 | TODO | Docs Guild, VEX Lens Guild | VEXLENS-30-005 | Author `/docs/vex/consensus-algorithm.md` covering normalization, weighting, thresholds, examples. | Doc merged; math reviewed by Policy; checklist appended. |
|
||||
| DOCS-VEX-30-003 | TODO | Docs Guild, Issuer Directory Guild | ISSUER-30-001..003 | Document `/docs/vex/issuer-directory.md` (issuer management, keys, trust overrides, audit). | Doc merged; security review done; checklist appended. |
|
||||
| DOCS-VEX-30-004 | TODO | Docs Guild, VEX Lens Guild | VEXLENS-30-007 | Publish `/docs/vex/consensus-api.md` with endpoint specs, query params, rate limits. | API doc aligned with OpenAPI; examples validated; checklist appended. |
|
||||
| DOCS-VEX-30-005 | TODO | Docs Guild, Console Guild | CONSOLE-VEX-30-001 | Write `/docs/vex/consensus-console.md` covering UI workflows, filters, conflicts, accessibility. | Doc merged; screenshots added; checklist appended. |
|
||||
| DOCS-VEX-30-006 | TODO | Docs Guild, Policy Guild | POLICY-ENGINE-29-001, VEXLENS-30-004 | Add `/docs/policy/vex-trust-model.md` detailing policy knobs, thresholds, simulation. | Doc merged; policy review completed; checklist appended. |
|
||||
| DOCS-VEX-30-007 | TODO | Docs Guild, SBOM Service Guild | VEXLENS-30-002 | Publish `/docs/sbom/vex-mapping.md` (CPE→purl strategy, edge cases, overrides). | Doc merged; mapping tables validated; checklist appended. |
|
||||
| DOCS-VEX-30-008 | TODO | Docs Guild, Security Guild | ISSUER-30-002, VEXLENS-30-003 | Deliver `/docs/security/vex-signatures.md` (verification flow, key rotation, audit). | Doc approved by Security; checklist appended. |
|
||||
| DOCS-VEX-30-009 | TODO | Docs Guild, DevOps Guild | VEXLENS-30-009, DEVOPS-VEX-30-001 | Create `/docs/runbooks/vex-ops.md` for recompute storms, mapping failures, signature errors. | Runbook merged; rehearsal logged; checklist appended. |
|
||||
|
||||
## Advisory AI (Sprint 31)
|
||||
|
||||
| ID | Status | Owner(s) | Depends on | Description | Exit Criteria |
|
||||
|----|--------|----------|------------|-------------|---------------|
|
||||
| DOCS-AIAI-31-001 | TODO | Docs Guild, Advisory AI Guild | AIAI-31-006 | Publish `/docs/advisory-ai/overview.md` covering capabilities, guardrails, RBAC. | Doc merged with diagrams; compliance checklist appended. |
|
||||
| DOCS-AIAI-31-002 | TODO | Docs Guild, Advisory AI Guild | AIAI-31-004 | Author `/docs/advisory-ai/architecture.md` detailing RAG pipeline, deterministics, caching, model options. | Doc merged; architecture review done; checklist appended. |
|
||||
| DOCS-AIAI-31-003 | TODO | Docs Guild, Advisory AI Guild | AIAI-31-006 | Write `/docs/advisory-ai/api.md` describing endpoints, schemas, errors, rate limits. | API doc aligned with OpenAPI; examples validated; checklist appended. |
|
||||
| DOCS-AIAI-31-004 | TODO | Docs Guild, Console Guild | CONSOLE-VULN-29-001, CONSOLE-VEX-30-001 | Create `/docs/advisory-ai/console.md` with screenshots, a11y notes, copy-as-ticket instructions. | Doc merged; images stored; checklist appended. |
|
||||
| DOCS-AIAI-31-005 | TODO | Docs Guild, DevEx/CLI Guild | CLI-VULN-29-001, CLI-VEX-30-001 | Publish `/docs/advisory-ai/cli.md` covering commands, exit codes, scripting patterns. | Doc merged; examples tested; checklist appended. |
|
||||
| DOCS-AIAI-31-006 | TODO | Docs Guild, Policy Guild | POLICY-ENGINE-31-001 | Update `/docs/policy/assistant-parameters.md` covering temperature, token limits, ranking weights, TTLs. | Doc merged; policy review done; checklist appended. |
|
||||
| DOCS-AIAI-31-007 | TODO | Docs Guild, Security Guild | AIAI-31-005 | Write `/docs/security/assistant-guardrails.md` detailing redaction, injection defense, logging. | Doc approved by Security; checklist appended. |
|
||||
| DOCS-AIAI-31-008 | TODO | Docs Guild, SBOM Service Guild | SBOM-AIAI-31-001 | Publish `/docs/sbom/remediation-heuristics.md` (feasibility scoring, blast radius). | Doc merged; heuristics reviewed; checklist appended. |
|
||||
| DOCS-AIAI-31-009 | TODO | Docs Guild, DevOps Guild | DEVOPS-AIAI-31-001 | Create `/docs/runbooks/assistant-ops.md` for warmup, cache priming, model outages, scaling. | Runbook merged; rehearsal logged; checklist appended. |
|
||||
|
||||
## Notifications Studio
|
||||
|
||||
| ID | Status | Owner(s) | Depends on | Description | Exit Criteria |
|
||||
|----|--------|----------|------------|-------------|---------------|
|
||||
| DOCS-NOTIFY-38-001 | DONE (2025-10-29) | Docs Guild, Notifications Service Guild | NOTIFY-SVC-38-001..004 | Publish `/docs/notifications/overview.md` and `/docs/notifications/architecture.md`, each ending with imposed rule reminder. | Docs merged; diagrams verified; imposed rule appended. |
|
||||
| DOCS-NOTIFY-39-002 | DONE (2025-10-29) | Docs Guild, Notifications Service Guild | NOTIFY-SVC-39-001..004 | Publish `/docs/notifications/rules.md`, `/docs/notifications/templates.md`, `/docs/notifications/digests.md` with examples and imposed rule line. | Docs merged; examples validated; imposed rule appended. |
|
||||
| DOCS-NOTIFY-40-001 | TODO | Docs Guild, Security Guild | AUTH-NOTIFY-38-001, NOTIFY-SVC-40-001..004 | Publish `/docs/notifications/channels.md`, `/docs/notifications/escalations.md`, `/docs/notifications/api.md`, `/docs/operations/notifier-runbook.md`, `/docs/security/notifications-hardening.md`; each ends with imposed rule line. | Docs merged; accessibility checks passed; imposed rule appended. |
|
||||
|
||||
## CLI Parity & Task Packs
|
||||
|
||||
| ID | Status | Owner(s) | Depends on | Description | Exit Criteria |
|
||||
|----|--------|----------|------------|-------------|---------------|
|
||||
| DOCS-CLI-41-001 | TODO | Docs Guild, DevEx/CLI Guild | CLI-CORE-41-001 | Publish `/docs/cli/overview.md`, `/docs/cli/configuration.md`, `/docs/cli/output-and-exit-codes.md` with imposed rule statements. | Docs merged; examples verified; imposed rule appended. |
|
||||
| DOCS-CLI-42-001 | TODO | Docs Guild | DOCS-CLI-41-001, CLI-PARITY-41-001 | Publish `/docs/cli/parity-matrix.md` and command guides under `/docs/cli/commands/*.md` (policy, sbom, vuln, vex, advisory, export, orchestrator, notify, aoc, auth). | Guides merged; parity automation documented; imposed rule appended. |
|
||||
| DOCS-PACKS-43-001 | DONE (2025-10-27) | Docs Guild, Task Runner Guild | PACKS-REG-42-001, TASKRUN-42-001 | Publish `/docs/task-packs/spec.md`, `/docs/task-packs/authoring-guide.md`, `/docs/task-packs/registry.md`, `/docs/task-packs/runbook.md`, `/docs/security/pack-signing-and-rbac.md`, `/docs/operations/cli-release-and-packaging.md` with imposed rule statements. | Docs merged; tutorials validated; imposed rule appended; cross-links added. |
|
||||
|
||||
## Containerized Distribution (Epic 13)
|
||||
|
||||
| ID | Status | Owner(s) | Depends on | Description | Exit Criteria |
|
||||
|----|--------|----------|------------|-------------|---------------|
|
||||
| DOCS-INSTALL-44-001 | TODO | Docs Guild, Deployment Guild | COMPOSE-44-001 | Publish `/docs/install/overview.md` and `/docs/install/compose-quickstart.md` with imposed rule line and copy-ready commands. | Docs merged; screenshots/commands verified; imposed rule appended. |
|
||||
| DOCS-INSTALL-45-001 | TODO | Docs Guild, Deployment Guild | HELM-45-001 | Publish `/docs/install/helm-prod.md` and `/docs/install/configuration-reference.md` with values tables and imposed rule reminder. | Docs merged; configuration matrix verified; imposed rule appended. |
|
||||
| DOCS-INSTALL-46-001 | TODO | Docs Guild, Security Guild | DEPLOY-PACKS-43-001, CLI-PACKS-43-001 | Publish `/docs/install/airgap.md`, `/docs/security/supply-chain.md`, `/docs/operations/health-and-readiness.md`, `/docs/release/image-catalog.md`, `/docs/console/onboarding.md` (each with imposed rule). | Docs merged; checksum/signature sections validated; imposed rule appended. |
|
||||
|
||||
## Authority-Backed Scopes & Tenancy (Epic 14)
|
||||
| ID | Status | Owner(s) | Depends on | Description | Exit Criteria |
|
||||
|----|--------|----------|------------|-------------|---------------|
|
||||
| DOCS-TEN-47-001 | TODO | Docs Guild, Authority Core | AUTH-TEN-47-001 | Publish `/docs/security/tenancy-overview.md` and `/docs/security/scopes-and-roles.md` outlining scope grammar, tenant model, imposed rule reminder. | Docs merged; diagrams included; imposed rule appended. |
|
||||
| DOCS-TEN-48-001 | TODO | Docs Guild, Platform Ops | WEB-TEN-48-001 | Publish `/docs/operations/multi-tenancy.md`, `/docs/operations/rls-and-data-isolation.md`, `/docs/console/admin-tenants.md`. | Docs merged; examples validated; imposed rule appended. |
|
||||
| DOCS-TEN-49-001 | TODO | Docs & DevEx Guilds | CLI-TEN-47-001, AUTH-TEN-49-001 | Publish `/docs/cli/authentication.md`, `/docs/api/authentication.md`, `/docs/policy/examples/abac-overlays.md`, update `/docs/install/configuration-reference.md` with new env vars, all ending with imposed rule line. | Docs merged; command examples verified; imposed rule appended. |
|
||||
|
||||
@@ -1,131 +1,131 @@
|
||||
# StellaOps Console Accessibility Guide
|
||||
|
||||
> **Audience:** Accessibility Guild, Console Guild, Docs Guild, QA.
|
||||
> **Scope:** Keyboard interaction model, screen-reader behaviour, colour & focus tokens, testing workflows, offline considerations, and compliance checklist for the StellaOps Console (Sprint 23).
|
||||
|
||||
The console targets **WCAG 2.2 AA** across all supported browsers (Chromium, Firefox ESR) and honours StellaOps’ sovereign/offline constraints. Every build must keep keyboard-only users, screen-reader users, and high-contrast operators productive without relying on third-party services.
|
||||
|
||||
---
|
||||
|
||||
## 1 · Accessibility Principles
|
||||
|
||||
1. **Deterministic navigation** – Focus order, shortcuts, and announcements remain stable across releases; URLs encode state for deep links.
|
||||
2. **Keyboard-first design** – Every actionable element is reachable via keyboard; shortcuts provide accelerators, and remapping is available via *Settings → Accessibility → Keyboard shortcuts*.
|
||||
3. **Assistive technology parity** – ARIA roles and live regions mirror visual affordances (status banners, SSE tickers, progress drawers). Screen readers receive polite/atomic updates to avoid chatter.
|
||||
4. **Colour & contrast tokens** – All palettes derive from design tokens that achieve ≥ 4.5:1 contrast (text) and ≥ 3:1 for graphical indicators; tokens pass automated contrast linting.
|
||||
5. **Offline equivalence** – Accessibility features (shortcuts, offline banners, focus restoration) behave the same in sealed environments, with guidance when actions require online authority.
|
||||
|
||||
---
|
||||
|
||||
## 2 · Keyboard Interaction Map
|
||||
|
||||
### 2.1 Global shortcuts
|
||||
|
||||
| Action | Macs | Windows/Linux | Notes |
|
||||
|--------|------|---------------|-------|
|
||||
| Command palette | `⌘ K` | `Ctrl K` | Focuses palette search; respects tenant scope. |
|
||||
| Tenant picker | `⌘ T` | `Ctrl T` | Opens modal; `Enter` confirms, `Esc` cancels. |
|
||||
| Filter tray toggle | `⇧ F` | `Shift F` | Focus lands on first filter; `Tab` cycles filters before returning to page. |
|
||||
| Saved view presets | `⌘ 1-9` | `Ctrl 1-9` | Bound per tenant; missing preset triggers tooltip. |
|
||||
| Keyboard reference | `?` | `?` | Opens overlay listing context-specific shortcuts; `Esc` closes. |
|
||||
| Global search (context) | `/` | `/` | When the filter tray is closed, focuses inline search field. |
|
||||
|
||||
### 2.2 Module-specific shortcuts
|
||||
|
||||
| Module | Action | Macs | Windows/Linux | Notes |
|
||||
|--------|--------|------|---------------|-------|
|
||||
| Findings | Explain search | `⌘ /` | `Ctrl /` | Only when Explain drawer open; announces results via live region. |
|
||||
| SBOM Explorer | Toggle overlays | `⌘ G` | `Ctrl G` | Persists per session (see `/docs/ui/sbom-explorer.md`). |
|
||||
| Advisories & VEX | Provider filter | `⌘ ⌥ F` | `Ctrl Alt F` | Moves focus to provider chip row. |
|
||||
| Runs | Refresh snapshot | `⌘ R` | `Ctrl R` | Soft refresh of SSE state; no full page reload. |
|
||||
| Policies | Save draft | `⌘ S` | `Ctrl S` | Requires edit scope; exposes toast + status live update. |
|
||||
| Downloads | Copy CLI command | `⇧ D` | `Shift D` | Copies manifest or export command; toast announces scope hints. |
|
||||
|
||||
All shortcuts are remappable. Remaps persist in IndexedDB (per tenant) and export as part of profile bundles so operators can restore preferences offline.
|
||||
|
||||
---
|
||||
|
||||
## 3 · Screen Reader & Focus Behaviour
|
||||
|
||||
- **Skip navigation** – Each route exposes a “Skip to content” link revealed on keyboard focus. Focus order: global header → page breadcrumb → action shelf → data grid/list → drawers/dialogs.
|
||||
- **Live regions** – Status ticker and SSE progress bars use `aria-live="polite"` with throttling to avoid flooding AT. Error toasts use `aria-live="assertive"` and auto-focus dismiss buttons.
|
||||
- **Drawers & modals** – Dialog components trap focus, support `Esc` to close, and restore focus to the launching control. Screen readers announce title + purpose.
|
||||
- **Tables & grids** – Large tables (Findings, SBOM inventory) switch to virtualised rows but retain ARIA grid semantics (`aria-rowcount`, `aria-colindex`). Column headers include sorting state via `aria-sort`.
|
||||
- **Tenancy context** – Tenant badge exposes `aria-describedby` linking to context summary (environment, offline snapshot). Switching tenant queues a polite announcement summarising new scope.
|
||||
- **Command palette** – Uses `role="dialog"` with search input labelled. Keyboard navigation within results uses `Up/Down`; screen readers announce result category + command.
|
||||
- **Offline banner** – When offline, a dismissible banner announces reason and includes instructions for CLI fallback. The banner has `role="status"` so it announces once without stealing focus.
|
||||
|
||||
---
|
||||
|
||||
## 4 · Colour & Focus Tokens
|
||||
|
||||
Console consumes design tokens published by the Console Guild (tracked via CONSOLE-FEAT-23-102). Tokens live in the design system bundle (`ui/design/tokens/colors.json`, mirrored at build time). Key tokens:
|
||||
|
||||
| Token | Purpose | Contrast target |
|
||||
|-------|---------|-----------------|
|
||||
| `so-color-surface-base` | Primary surface/background | ≥ 4.5:1 against `so-color-text-primary`. |
|
||||
| `so-color-surface-raised` | Cards, drawers, modals | ≥ 3:1 against surrounding surfaces. |
|
||||
| `so-color-text-primary` | Default text colour | ≥ 4.5:1 against base surfaces. |
|
||||
| `so-color-text-inverted` | Text on accent buttons | ≥ 4.5:1 against accent fills. |
|
||||
| `so-color-accent-primary` | Action buttons, focus headings | ≥ 3:1 against surface. |
|
||||
| `so-color-status-critical` | Error toasts, violation chips | ≥ 4.5:1 for text; `critical-bg` provides >3:1 on neutral surface. |
|
||||
| `so-color-status-warning` | Warning banners | Meets 3:1 on surface and 4.5:1 for text overlays. |
|
||||
| `so-color-status-success` | Success toasts, pass badges | ≥ 3:1 for iconography; text uses `text-primary`. |
|
||||
| `so-focus-ring` | 2 px outline used across focusable elements | 3:1 against both light/dark surfaces. |
|
||||
|
||||
Colour tokens undergo automated linting (**axe-core contrast checks** + custom luminance script) during build. Any new token must include dark/light variants and pass the token contract tests.
|
||||
|
||||
---
|
||||
|
||||
## 5 · Testing Workflow
|
||||
|
||||
| Layer | Tooling | Frequency | Notes |
|
||||
|-------|---------|-----------|-------|
|
||||
| Component a11y | Storybook + axe-core addon | On PR (story CI) | Fails when axe detects violations. |
|
||||
| Route regression | Playwright a11y sweep (`pnpm test:a11y`) | Nightly & release pipeline | Executes keyboard navigation, checks focus trap, runs Axe on key routes (Dashboard, Findings, SBOM, Admin). |
|
||||
| Colour contrast lint | Token validator (`tools/a11y/check-contrast.ts`) | On token change | Guards design token updates. |
|
||||
| CI parity | Pending `scripts/check-console-cli-parity.sh` (CONSOLE-DOC-23-502) | Release CI | Ensures CLI commands documented for parity features. |
|
||||
| Screen-reader spot checks | Manual NVDA + VoiceOver scripts | Pre-release checklist | Scenarios: tenant switch, explain drawer, downloads parity copy. |
|
||||
| Offline smoke | `stella offline kit import` + Playwright sealed-mode run | Prior to Offline Kit cut | Validates offline banners, disabled actions, keyboard flows without Authority. |
|
||||
|
||||
Accessibility QA (CONSOLE-QA-23-402) tracks failing scenarios via Playwright snapshots and publishes reports in the Downloads parity channel (`kind = "parity.report"` placeholder until CLI parity CI lands).
|
||||
|
||||
---
|
||||
|
||||
## 6 · Offline & Internationalisation Considerations
|
||||
|
||||
- Offline mode surfaces staleness badges and disables remote-only palette entries; keyboard focus skips disabled controls.
|
||||
- Saved shortcuts, presets, and remaps serialise into Offline Kit bundles so operators can restore preferences post-import.
|
||||
- Locale switching (future feature flag) will load translations at runtime; ensure ARIA labels use i18n tokens rather than hard-coded strings.
|
||||
- For sealed installs, guidance panels include CLI equivalents (`stella auth fresh-auth`, `stella runs export`) to unblock tasks when Authority is unavailable.
|
||||
|
||||
---
|
||||
|
||||
## 7 · Compliance Checklist
|
||||
|
||||
- [ ] Keyboard shortcut matrix validated (default + remapped) and documented.
|
||||
- [ ] Screen-reader pass recorded for tenant switch, Explain drawer, Downloads copy-to-clipboard.
|
||||
- [ ] Colour tokens audited; contrast reports stored with release artifacts.
|
||||
- [ ] Automated a11y pipelines (Storybook axe, Playwright a11y) green; failures feed the `#console-qa` channel.
|
||||
- [ ] Offline kit a11y smoke executed before publishing each bundle.
|
||||
- [ ] CLI parity gaps logged in `/docs/cli-vs-ui-parity.md`; UI callouts reference fallback commands until parity closes.
|
||||
- [ ] Accessibility Guild sign-off captured in sprint log and release notes reference this guide.
|
||||
- [ ] References cross-checked (`/docs/ui/navigation.md`, `/docs/ui/downloads.md`, `/docs/security/console-security.md`, `/docs/observability/ui-telemetry.md`).
|
||||
|
||||
---
|
||||
|
||||
## 8 · References
|
||||
|
||||
- `/docs/ui/navigation.md` – shortcut definitions, URL schema.
|
||||
- `/docs/ui/downloads.md` – CLI parity and offline copy workflows.
|
||||
- `/docs/ui/console-overview.md` – tenant model, filter behaviours.
|
||||
- `/docs/security/console-security.md` – security metrics and DPoP/fresh-auth requirements.
|
||||
- `/docs/observability/ui-telemetry.md` – telemetry metrics mapped to accessibility features.
|
||||
- `/docs/cli-vs-ui-parity.md` – parity status per console feature.
|
||||
- `CONSOLE-QA-23-402` – Accessibility QA backlog (Playwright + manual checks).
|
||||
- `CONSOLE-FEAT-23-102` – Design tokens & theming delivery.
|
||||
|
||||
---
|
||||
|
||||
*Last updated: 2025-10-28 (Sprint 23).*
|
||||
|
||||
# StellaOps Console Accessibility Guide
|
||||
|
||||
> **Audience:** Accessibility Guild, Console Guild, Docs Guild, QA.
|
||||
> **Scope:** Keyboard interaction model, screen-reader behaviour, colour & focus tokens, testing workflows, offline considerations, and compliance checklist for the StellaOps Console (Sprint 23).
|
||||
|
||||
The console targets **WCAG 2.2 AA** across all supported browsers (Chromium, Firefox ESR) and honours StellaOps’ sovereign/offline constraints. Every build must keep keyboard-only users, screen-reader users, and high-contrast operators productive without relying on third-party services.
|
||||
|
||||
---
|
||||
|
||||
## 1 · Accessibility Principles
|
||||
|
||||
1. **Deterministic navigation** – Focus order, shortcuts, and announcements remain stable across releases; URLs encode state for deep links.
|
||||
2. **Keyboard-first design** – Every actionable element is reachable via keyboard; shortcuts provide accelerators, and remapping is available via *Settings → Accessibility → Keyboard shortcuts*.
|
||||
3. **Assistive technology parity** – ARIA roles and live regions mirror visual affordances (status banners, SSE tickers, progress drawers). Screen readers receive polite/atomic updates to avoid chatter.
|
||||
4. **Colour & contrast tokens** – All palettes derive from design tokens that achieve ≥ 4.5:1 contrast (text) and ≥ 3:1 for graphical indicators; tokens pass automated contrast linting.
|
||||
5. **Offline equivalence** – Accessibility features (shortcuts, offline banners, focus restoration) behave the same in sealed environments, with guidance when actions require online authority.
|
||||
|
||||
---
|
||||
|
||||
## 2 · Keyboard Interaction Map
|
||||
|
||||
### 2.1 Global shortcuts
|
||||
|
||||
| Action | Macs | Windows/Linux | Notes |
|
||||
|--------|------|---------------|-------|
|
||||
| Command palette | `⌘ K` | `Ctrl K` | Focuses palette search; respects tenant scope. |
|
||||
| Tenant picker | `⌘ T` | `Ctrl T` | Opens modal; `Enter` confirms, `Esc` cancels. |
|
||||
| Filter tray toggle | `⇧ F` | `Shift F` | Focus lands on first filter; `Tab` cycles filters before returning to page. |
|
||||
| Saved view presets | `⌘ 1-9` | `Ctrl 1-9` | Bound per tenant; missing preset triggers tooltip. |
|
||||
| Keyboard reference | `?` | `?` | Opens overlay listing context-specific shortcuts; `Esc` closes. |
|
||||
| Global search (context) | `/` | `/` | When the filter tray is closed, focuses inline search field. |
|
||||
|
||||
### 2.2 Module-specific shortcuts
|
||||
|
||||
| Module | Action | Macs | Windows/Linux | Notes |
|
||||
|--------|--------|------|---------------|-------|
|
||||
| Findings | Explain search | `⌘ /` | `Ctrl /` | Only when Explain drawer open; announces results via live region. |
|
||||
| SBOM Explorer | Toggle overlays | `⌘ G` | `Ctrl G` | Persists per session (see `/docs/ui/sbom-explorer.md`). |
|
||||
| Advisories & VEX | Provider filter | `⌘ ⌥ F` | `Ctrl Alt F` | Moves focus to provider chip row. |
|
||||
| Runs | Refresh snapshot | `⌘ R` | `Ctrl R` | Soft refresh of SSE state; no full page reload. |
|
||||
| Policies | Save draft | `⌘ S` | `Ctrl S` | Requires edit scope; exposes toast + status live update. |
|
||||
| Downloads | Copy CLI command | `⇧ D` | `Shift D` | Copies manifest or export command; toast announces scope hints. |
|
||||
|
||||
All shortcuts are remappable. Remaps persist in IndexedDB (per tenant) and export as part of profile bundles so operators can restore preferences offline.
|
||||
|
||||
---
|
||||
|
||||
## 3 · Screen Reader & Focus Behaviour
|
||||
|
||||
- **Skip navigation** – Each route exposes a “Skip to content” link revealed on keyboard focus. Focus order: global header → page breadcrumb → action shelf → data grid/list → drawers/dialogs.
|
||||
- **Live regions** – Status ticker and SSE progress bars use `aria-live="polite"` with throttling to avoid flooding AT. Error toasts use `aria-live="assertive"` and auto-focus dismiss buttons.
|
||||
- **Drawers & modals** – Dialog components trap focus, support `Esc` to close, and restore focus to the launching control. Screen readers announce title + purpose.
|
||||
- **Tables & grids** – Large tables (Findings, SBOM inventory) switch to virtualised rows but retain ARIA grid semantics (`aria-rowcount`, `aria-colindex`). Column headers include sorting state via `aria-sort`.
|
||||
- **Tenancy context** – Tenant badge exposes `aria-describedby` linking to context summary (environment, offline snapshot). Switching tenant queues a polite announcement summarising new scope.
|
||||
- **Command palette** – Uses `role="dialog"` with search input labelled. Keyboard navigation within results uses `Up/Down`; screen readers announce result category + command.
|
||||
- **Offline banner** – When offline, a dismissible banner announces reason and includes instructions for CLI fallback. The banner has `role="status"` so it announces once without stealing focus.
|
||||
|
||||
---
|
||||
|
||||
## 4 · Colour & Focus Tokens
|
||||
|
||||
Console consumes design tokens published by the Console Guild (tracked via CONSOLE-FEAT-23-102). Tokens live in the design system bundle (`ui/design/tokens/colors.json`, mirrored at build time). Key tokens:
|
||||
|
||||
| Token | Purpose | Contrast target |
|
||||
|-------|---------|-----------------|
|
||||
| `so-color-surface-base` | Primary surface/background | ≥ 4.5:1 against `so-color-text-primary`. |
|
||||
| `so-color-surface-raised` | Cards, drawers, modals | ≥ 3:1 against surrounding surfaces. |
|
||||
| `so-color-text-primary` | Default text colour | ≥ 4.5:1 against base surfaces. |
|
||||
| `so-color-text-inverted` | Text on accent buttons | ≥ 4.5:1 against accent fills. |
|
||||
| `so-color-accent-primary` | Action buttons, focus headings | ≥ 3:1 against surface. |
|
||||
| `so-color-status-critical` | Error toasts, violation chips | ≥ 4.5:1 for text; `critical-bg` provides >3:1 on neutral surface. |
|
||||
| `so-color-status-warning` | Warning banners | Meets 3:1 on surface and 4.5:1 for text overlays. |
|
||||
| `so-color-status-success` | Success toasts, pass badges | ≥ 3:1 for iconography; text uses `text-primary`. |
|
||||
| `so-focus-ring` | 2 px outline used across focusable elements | 3:1 against both light/dark surfaces. |
|
||||
|
||||
Colour tokens undergo automated linting (**axe-core contrast checks** + custom luminance script) during build. Any new token must include dark/light variants and pass the token contract tests.
|
||||
|
||||
---
|
||||
|
||||
## 5 · Testing Workflow
|
||||
|
||||
| Layer | Tooling | Frequency | Notes |
|
||||
|-------|---------|-----------|-------|
|
||||
| Component a11y | Storybook + axe-core addon | On PR (story CI) | Fails when axe detects violations. |
|
||||
| Route regression | Playwright a11y sweep (`pnpm test:a11y`) | Nightly & release pipeline | Executes keyboard navigation, checks focus trap, runs Axe on key routes (Dashboard, Findings, SBOM, Admin). |
|
||||
| Colour contrast lint | Token validator (`tools/a11y/check-contrast.ts`) | On token change | Guards design token updates. |
|
||||
| CI parity | Pending `scripts/check-console-cli-parity.sh` (CONSOLE-DOC-23-502) | Release CI | Ensures CLI commands documented for parity features. |
|
||||
| Screen-reader spot checks | Manual NVDA + VoiceOver scripts | Pre-release checklist | Scenarios: tenant switch, explain drawer, downloads parity copy. |
|
||||
| Offline smoke | `stella offline kit import` + Playwright sealed-mode run | Prior to Offline Kit cut | Validates offline banners, disabled actions, keyboard flows without Authority. |
|
||||
|
||||
Accessibility QA (CONSOLE-QA-23-402) tracks failing scenarios via Playwright snapshots and publishes reports in the Downloads parity channel (`kind = "parity.report"` placeholder until CLI parity CI lands).
|
||||
|
||||
---
|
||||
|
||||
## 6 · Offline & Internationalisation Considerations
|
||||
|
||||
- Offline mode surfaces staleness badges and disables remote-only palette entries; keyboard focus skips disabled controls.
|
||||
- Saved shortcuts, presets, and remaps serialise into Offline Kit bundles so operators can restore preferences post-import.
|
||||
- Locale switching (future feature flag) will load translations at runtime; ensure ARIA labels use i18n tokens rather than hard-coded strings.
|
||||
- For sealed installs, guidance panels include CLI equivalents (`stella auth fresh-auth`, `stella runs export`) to unblock tasks when Authority is unavailable.
|
||||
|
||||
---
|
||||
|
||||
## 7 · Compliance Checklist
|
||||
|
||||
- [ ] Keyboard shortcut matrix validated (default + remapped) and documented.
|
||||
- [ ] Screen-reader pass recorded for tenant switch, Explain drawer, Downloads copy-to-clipboard.
|
||||
- [ ] Colour tokens audited; contrast reports stored with release artifacts.
|
||||
- [ ] Automated a11y pipelines (Storybook axe, Playwright a11y) green; failures feed the `#console-qa` channel.
|
||||
- [ ] Offline kit a11y smoke executed before publishing each bundle.
|
||||
- [ ] CLI parity gaps logged in `/docs/cli-vs-ui-parity.md`; UI callouts reference fallback commands until parity closes.
|
||||
- [ ] Accessibility Guild sign-off captured in sprint log and release notes reference this guide.
|
||||
- [ ] References cross-checked (`/docs/ui/navigation.md`, `/docs/ui/downloads.md`, `/docs/security/console-security.md`, `/docs/observability/ui-telemetry.md`).
|
||||
|
||||
---
|
||||
|
||||
## 8 · References
|
||||
|
||||
- `/docs/ui/navigation.md` – shortcut definitions, URL schema.
|
||||
- `/docs/ui/downloads.md` – CLI parity and offline copy workflows.
|
||||
- `/docs/ui/console-overview.md` – tenant model, filter behaviours.
|
||||
- `/docs/security/console-security.md` – security metrics and DPoP/fresh-auth requirements.
|
||||
- `/docs/observability/ui-telemetry.md` – telemetry metrics mapped to accessibility features.
|
||||
- `/docs/cli-vs-ui-parity.md` – parity status per console feature.
|
||||
- `CONSOLE-QA-23-402` – Accessibility QA backlog (Playwright + manual checks).
|
||||
- `CONSOLE-FEAT-23-102` – Design tokens & theming delivery.
|
||||
|
||||
---
|
||||
|
||||
*Last updated: 2025-10-28 (Sprint 23).*
|
||||
|
||||
|
||||
@@ -1,218 +1,218 @@
|
||||
# Advisory Observations & Linksets
|
||||
|
||||
> Imposed rule: Work of this type or tasks of this type on this component must also
|
||||
> be applied everywhere else it should be applied.
|
||||
|
||||
The Link-Not-Merge (LNM) initiative replaces the legacy "merge" pipeline with
|
||||
immutable observations and correlation linksets. This guide explains how
|
||||
Concelier ingests advisory statements, preserves upstream truth, and produces
|
||||
linksets that downstream services (Policy Engine, Vuln Explorer, Console) can
|
||||
use without collapsing sources together.
|
||||
|
||||
---
|
||||
|
||||
## 1. Model overview
|
||||
|
||||
### 1.1 Observation lifecycle
|
||||
|
||||
1. **Ingest** – Connectors fetch upstream payloads (CSAF, OSV, vendor feeds),
|
||||
validate signatures, and drop any derived fields prohibited by the
|
||||
Aggregation-Only Contract (AOC).
|
||||
2. **Persist** – Concelier writes immutable `advisory_observations` scoped by
|
||||
`tenant`, `(source.vendor, upstreamId)`, and `contentHash`. Supersedes chains
|
||||
capture revisions without mutating history.
|
||||
3. **Expose** – WebService surfaces paged/read APIs; Offline Kit snapshots
|
||||
include the same documents for air-gapped installs.
|
||||
|
||||
Observation schema highlights:
|
||||
|
||||
```text
|
||||
observationId = {tenant}:{source.vendor}:{upstreamId}:{revision}
|
||||
tenant, source{vendor, stream, api, collectorVersion}
|
||||
upstream{upstreamId, documentVersion, fetchedAt, receivedAt,
|
||||
contentHash, signature{present, format, keyId, signature}}
|
||||
content{format, specVersion, raw}
|
||||
identifiers{cve?, ghsa?, aliases[], osvIds[]}
|
||||
linkset{purls[], cpes[], aliases[], references[], conflicts[]?}
|
||||
createdAt, attributes{batchId?, replayCursor?}
|
||||
```
|
||||
|
||||
- **Immutable raw** (`content.raw`) mirrors upstream payloads exactly.
|
||||
- **Provenance** (`source.*`, `upstream.*`) satisfies AOC guardrails and enables
|
||||
cryptographic attestations.
|
||||
- **Identifiers** retain lossless extracts (CVE, GHSA, vendor aliases) that seed
|
||||
linksets.
|
||||
- **Linkset** captures join hints but never merges or adds derived severity.
|
||||
|
||||
### 1.2 Linkset lifecycle
|
||||
|
||||
Linksets correlate observations that describe the same vulnerable product while
|
||||
keeping each source intact.
|
||||
|
||||
1. **Seed** – Observations emit normalized identifiers (`purl`, `cpe`,
|
||||
`alias`) during ingestion.
|
||||
2. **Correlate** – Linkset builder groups observations by tenant, product
|
||||
coordinates, and equivalence signals (PURL alias graph, CVE overlap, CVSS
|
||||
vector equality, fuzzy titles).
|
||||
3. **Annotate** – Detected conflicts (severity disagreements, affected-range
|
||||
mismatch, incompatible references) are recorded with structured payloads and
|
||||
preserved for UI/API export.
|
||||
4. **Persist** – Results land in `advisory_linksets` with deterministic IDs
|
||||
(`linksetId = {tenant}:{hash(aliases+purls+seedIds)}`) and append-only history
|
||||
for reproducibility.
|
||||
|
||||
Linksets never suppress or prefer one source; they provide aligned evidence so
|
||||
other services can apply policy.
|
||||
|
||||
---
|
||||
|
||||
## 2. Observation vs. linkset
|
||||
|
||||
- **Purpose**
|
||||
- Observation: Immutable record per vendor and revision.
|
||||
- Linkset: Correlates observations that share product identity.
|
||||
- **Mutation**
|
||||
- Observation: Append-only via supersedes chain.
|
||||
- Linkset: Rebuilt deterministically from canonical signals.
|
||||
- **Allowed fields**
|
||||
- Observation: Raw payload, provenance, identifiers, join hints.
|
||||
- Linkset: Observation references, normalized product metadata, conflicts.
|
||||
- **Forbidden fields**
|
||||
- Observation: Derived severity, policy status, opinionated dedupe.
|
||||
- Linkset: Derived severity (conflicts recorded but unresolved).
|
||||
- **Consumers**
|
||||
- Observation: Evidence API, Offline Kit, CLI exports.
|
||||
- Linkset: Policy Engine overlay, UI evidence panel, Vuln Explorer.
|
||||
|
||||
### 2.1 Example sequence
|
||||
|
||||
1. Red Hat PSIRT publishes RHSA-2025:1234 for OpenSSL; Concelier inserts an
|
||||
observation for vendor `redhat` with `pkg:rpm/redhat/openssl@1.1.1w-12`.
|
||||
2. NVD issues CVE-2025-0001; a second observation is inserted for vendor `nvd`.
|
||||
3. Linkset builder runs, groups the two observations, records alias and PURL
|
||||
overlap, and flags a CVSS disagreement (`7.5` vs `7.2`).
|
||||
4. Policy Engine reads the linkset, recognises the severity variance, and relies
|
||||
on configured rules to decide the effective output.
|
||||
|
||||
---
|
||||
|
||||
## 3. Conflict handling
|
||||
|
||||
Conflicts record disagreements without altering source payloads. The builder
|
||||
emits structured entries:
|
||||
|
||||
```json
|
||||
{
|
||||
"type": "severity-mismatch",
|
||||
"field": "cvss.baseScore",
|
||||
"observations": [
|
||||
{
|
||||
"source": "redhat",
|
||||
"value": "7.5",
|
||||
"vector": "AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:N/A:N"
|
||||
},
|
||||
{
|
||||
"source": "nvd",
|
||||
"value": "7.2",
|
||||
"vector": "AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:H/A:N"
|
||||
}
|
||||
],
|
||||
"confidence": "medium",
|
||||
"detectedAt": "2025-10-27T14:00:00Z"
|
||||
}
|
||||
```
|
||||
|
||||
Supported conflict classes:
|
||||
|
||||
- `severity-mismatch` – CVSS or qualitative severities differ.
|
||||
- `affected-range-divergence` – Product ranges, fixed versions, or platforms
|
||||
disagree.
|
||||
- `statement-disagreement` – One observation declares `not_affected` while
|
||||
another states `affected`.
|
||||
- `reference-clash` – URL or classifier collisions (for example, exploit URL vs
|
||||
conflicting advisory).
|
||||
- `alias-inconsistency` – Aliases map to different canonical IDs (GHSA vs CVE).
|
||||
- `metadata-gap` – Required provenance missing on one source; logged as a
|
||||
warning.
|
||||
|
||||
Conflict surfaces:
|
||||
|
||||
- WebService endpoints (`GET /advisories/linksets/{id}` → `conflicts[]`).
|
||||
- UI evidence panel chips and conflict badges.
|
||||
- CLI exports (JSON/OSV) exposed through LNM commands.
|
||||
- Observability metrics (`advisory_linkset_conflicts_total{type}`).
|
||||
|
||||
---
|
||||
|
||||
## 4. AOC alignment
|
||||
|
||||
Observations and linksets must satisfy Aggregation-Only Contract invariants:
|
||||
|
||||
- **No derived severity** – `content.raw` may include upstream severity, but the
|
||||
observation body never injects or edits severity.
|
||||
- **No merges** – Each upstream document stays separate; linksets reference
|
||||
observations via deterministic IDs.
|
||||
- **Provenance mandatory** – Missing `signature` or `source` metadata is an AOC
|
||||
violation (`ERR_AOC_004`).
|
||||
- **Idempotent writes** – Duplicate `contentHash` yields a no-op; supersedes
|
||||
pointer captures new revisions.
|
||||
- **Deterministic output** – Linkset builder sorts keys, normalizes timestamps
|
||||
(UTC ISO-8601), and uses canonical JSON hashing.
|
||||
|
||||
Violations trigger guard errors (`ERR_AOC_00x`), emit `aoc_violation_total`
|
||||
metrics, and block persistence until corrected.
|
||||
|
||||
---
|
||||
|
||||
## 5. Downstream consumption
|
||||
|
||||
- **Policy Engine** – Computes effective severity and risk overlays from linkset
|
||||
evidence and conflicts.
|
||||
- **Console UI** – Renders per-source statements, signed hashes, and conflict
|
||||
banners inside the evidence panel.
|
||||
- **CLI (`stella advisories linkset …`)** – Exports observations and linksets as
|
||||
JSON or OSV for offline triage.
|
||||
- **Offline Kit** – Shipping snapshots include observation and linkset
|
||||
collections for air-gap parity.
|
||||
- **Observability** – Dashboards track ingestion latency, conflict counts, and
|
||||
supersedes depth.
|
||||
|
||||
When adding new consumers, ensure they honour append-only semantics and do not
|
||||
mutate observation or linkset collections.
|
||||
|
||||
---
|
||||
|
||||
## 6. Validation & testing
|
||||
|
||||
- **Unit tests** (`StellaOps.Concelier.Core.Tests`) validate schema guards,
|
||||
deterministic linkset hashing, conflict detection fixtures, and supersedes
|
||||
chains.
|
||||
- **Mongo integration tests** (`StellaOps.Concelier.Storage.Mongo.Tests`) verify
|
||||
indexes and idempotent writes under concurrency.
|
||||
- **CLI smoke suites** confirm `stella advisories observations` and `stella
|
||||
advisories linksets` export stable JSON.
|
||||
- **Determinism checks** replay identical upstream payloads and assert that the
|
||||
resulting observation and linkset documents match byte for byte.
|
||||
- **Offline kit verification** simulates air-gapped bootstrap to confirm that
|
||||
snapshots align with live data.
|
||||
|
||||
Add fixtures whenever a new conflict type or correlation signal is introduced.
|
||||
Ensure canonical JSON serialization remains stable across .NET runtime updates.
|
||||
|
||||
---
|
||||
|
||||
## 7. Reviewer checklist
|
||||
|
||||
- Observation schema segment matches the latest `StellaOps.Concelier.Models`
|
||||
contract.
|
||||
- Linkset lifecycle covers correlation signals, conflict classes, and
|
||||
deterministic IDs.
|
||||
- AOC invariants are explicitly called out with violation codes.
|
||||
- Examples include multi-source correlation plus conflict annotation.
|
||||
- Downstream consumer guidance reflects active APIs and CLI features.
|
||||
- Testing section lists required suites (Core, Storage, CLI, Offline).
|
||||
- Imposed rule reminder is present at the top of the document.
|
||||
|
||||
Confirmed against Concelier Link-Not-Merge tasks:
|
||||
`CONCELIER-LNM-21-001..005`, `CONCELIER-LNM-21-101..103`,
|
||||
`CONCELIER-LNM-21-201..203`.
|
||||
# Advisory Observations & Linksets
|
||||
|
||||
> Imposed rule: Work of this type or tasks of this type on this component must also
|
||||
> be applied everywhere else it should be applied.
|
||||
|
||||
The Link-Not-Merge (LNM) initiative replaces the legacy "merge" pipeline with
|
||||
immutable observations and correlation linksets. This guide explains how
|
||||
Concelier ingests advisory statements, preserves upstream truth, and produces
|
||||
linksets that downstream services (Policy Engine, Vuln Explorer, Console) can
|
||||
use without collapsing sources together.
|
||||
|
||||
---
|
||||
|
||||
## 1. Model overview
|
||||
|
||||
### 1.1 Observation lifecycle
|
||||
|
||||
1. **Ingest** – Connectors fetch upstream payloads (CSAF, OSV, vendor feeds),
|
||||
validate signatures, and drop any derived fields prohibited by the
|
||||
Aggregation-Only Contract (AOC).
|
||||
2. **Persist** – Concelier writes immutable `advisory_observations` scoped by
|
||||
`tenant`, `(source.vendor, upstreamId)`, and `contentHash`. Supersedes chains
|
||||
capture revisions without mutating history.
|
||||
3. **Expose** – WebService surfaces paged/read APIs; Offline Kit snapshots
|
||||
include the same documents for air-gapped installs.
|
||||
|
||||
Observation schema highlights:
|
||||
|
||||
```text
|
||||
observationId = {tenant}:{source.vendor}:{upstreamId}:{revision}
|
||||
tenant, source{vendor, stream, api, collectorVersion}
|
||||
upstream{upstreamId, documentVersion, fetchedAt, receivedAt,
|
||||
contentHash, signature{present, format, keyId, signature}}
|
||||
content{format, specVersion, raw}
|
||||
identifiers{cve?, ghsa?, aliases[], osvIds[]}
|
||||
linkset{purls[], cpes[], aliases[], references[], conflicts[]?}
|
||||
createdAt, attributes{batchId?, replayCursor?}
|
||||
```
|
||||
|
||||
- **Immutable raw** (`content.raw`) mirrors upstream payloads exactly.
|
||||
- **Provenance** (`source.*`, `upstream.*`) satisfies AOC guardrails and enables
|
||||
cryptographic attestations.
|
||||
- **Identifiers** retain lossless extracts (CVE, GHSA, vendor aliases) that seed
|
||||
linksets.
|
||||
- **Linkset** captures join hints but never merges or adds derived severity.
|
||||
|
||||
### 1.2 Linkset lifecycle
|
||||
|
||||
Linksets correlate observations that describe the same vulnerable product while
|
||||
keeping each source intact.
|
||||
|
||||
1. **Seed** – Observations emit normalized identifiers (`purl`, `cpe`,
|
||||
`alias`) during ingestion.
|
||||
2. **Correlate** – Linkset builder groups observations by tenant, product
|
||||
coordinates, and equivalence signals (PURL alias graph, CVE overlap, CVSS
|
||||
vector equality, fuzzy titles).
|
||||
3. **Annotate** – Detected conflicts (severity disagreements, affected-range
|
||||
mismatch, incompatible references) are recorded with structured payloads and
|
||||
preserved for UI/API export.
|
||||
4. **Persist** – Results land in `advisory_linksets` with deterministic IDs
|
||||
(`linksetId = {tenant}:{hash(aliases+purls+seedIds)}`) and append-only history
|
||||
for reproducibility.
|
||||
|
||||
Linksets never suppress or prefer one source; they provide aligned evidence so
|
||||
other services can apply policy.
|
||||
|
||||
---
|
||||
|
||||
## 2. Observation vs. linkset
|
||||
|
||||
- **Purpose**
|
||||
- Observation: Immutable record per vendor and revision.
|
||||
- Linkset: Correlates observations that share product identity.
|
||||
- **Mutation**
|
||||
- Observation: Append-only via supersedes chain.
|
||||
- Linkset: Rebuilt deterministically from canonical signals.
|
||||
- **Allowed fields**
|
||||
- Observation: Raw payload, provenance, identifiers, join hints.
|
||||
- Linkset: Observation references, normalized product metadata, conflicts.
|
||||
- **Forbidden fields**
|
||||
- Observation: Derived severity, policy status, opinionated dedupe.
|
||||
- Linkset: Derived severity (conflicts recorded but unresolved).
|
||||
- **Consumers**
|
||||
- Observation: Evidence API, Offline Kit, CLI exports.
|
||||
- Linkset: Policy Engine overlay, UI evidence panel, Vuln Explorer.
|
||||
|
||||
### 2.1 Example sequence
|
||||
|
||||
1. Red Hat PSIRT publishes RHSA-2025:1234 for OpenSSL; Concelier inserts an
|
||||
observation for vendor `redhat` with `pkg:rpm/redhat/openssl@1.1.1w-12`.
|
||||
2. NVD issues CVE-2025-0001; a second observation is inserted for vendor `nvd`.
|
||||
3. Linkset builder runs, groups the two observations, records alias and PURL
|
||||
overlap, and flags a CVSS disagreement (`7.5` vs `7.2`).
|
||||
4. Policy Engine reads the linkset, recognises the severity variance, and relies
|
||||
on configured rules to decide the effective output.
|
||||
|
||||
---
|
||||
|
||||
## 3. Conflict handling
|
||||
|
||||
Conflicts record disagreements without altering source payloads. The builder
|
||||
emits structured entries:
|
||||
|
||||
```json
|
||||
{
|
||||
"type": "severity-mismatch",
|
||||
"field": "cvss.baseScore",
|
||||
"observations": [
|
||||
{
|
||||
"source": "redhat",
|
||||
"value": "7.5",
|
||||
"vector": "AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:N/A:N"
|
||||
},
|
||||
{
|
||||
"source": "nvd",
|
||||
"value": "7.2",
|
||||
"vector": "AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:H/A:N"
|
||||
}
|
||||
],
|
||||
"confidence": "medium",
|
||||
"detectedAt": "2025-10-27T14:00:00Z"
|
||||
}
|
||||
```
|
||||
|
||||
Supported conflict classes:
|
||||
|
||||
- `severity-mismatch` – CVSS or qualitative severities differ.
|
||||
- `affected-range-divergence` – Product ranges, fixed versions, or platforms
|
||||
disagree.
|
||||
- `statement-disagreement` – One observation declares `not_affected` while
|
||||
another states `affected`.
|
||||
- `reference-clash` – URL or classifier collisions (for example, exploit URL vs
|
||||
conflicting advisory).
|
||||
- `alias-inconsistency` – Aliases map to different canonical IDs (GHSA vs CVE).
|
||||
- `metadata-gap` – Required provenance missing on one source; logged as a
|
||||
warning.
|
||||
|
||||
Conflict surfaces:
|
||||
|
||||
- WebService endpoints (`GET /advisories/linksets/{id}` → `conflicts[]`).
|
||||
- UI evidence panel chips and conflict badges.
|
||||
- CLI exports (JSON/OSV) exposed through LNM commands.
|
||||
- Observability metrics (`advisory_linkset_conflicts_total{type}`).
|
||||
|
||||
---
|
||||
|
||||
## 4. AOC alignment
|
||||
|
||||
Observations and linksets must satisfy Aggregation-Only Contract invariants:
|
||||
|
||||
- **No derived severity** – `content.raw` may include upstream severity, but the
|
||||
observation body never injects or edits severity.
|
||||
- **No merges** – Each upstream document stays separate; linksets reference
|
||||
observations via deterministic IDs.
|
||||
- **Provenance mandatory** – Missing `signature` or `source` metadata is an AOC
|
||||
violation (`ERR_AOC_004`).
|
||||
- **Idempotent writes** – Duplicate `contentHash` yields a no-op; supersedes
|
||||
pointer captures new revisions.
|
||||
- **Deterministic output** – Linkset builder sorts keys, normalizes timestamps
|
||||
(UTC ISO-8601), and uses canonical JSON hashing.
|
||||
|
||||
Violations trigger guard errors (`ERR_AOC_00x`), emit `aoc_violation_total`
|
||||
metrics, and block persistence until corrected.
|
||||
|
||||
---
|
||||
|
||||
## 5. Downstream consumption
|
||||
|
||||
- **Policy Engine** – Computes effective severity and risk overlays from linkset
|
||||
evidence and conflicts.
|
||||
- **Console UI** – Renders per-source statements, signed hashes, and conflict
|
||||
banners inside the evidence panel.
|
||||
- **CLI (`stella advisories linkset …`)** – Exports observations and linksets as
|
||||
JSON or OSV for offline triage.
|
||||
- **Offline Kit** – Shipping snapshots include observation and linkset
|
||||
collections for air-gap parity.
|
||||
- **Observability** – Dashboards track ingestion latency, conflict counts, and
|
||||
supersedes depth.
|
||||
|
||||
When adding new consumers, ensure they honour append-only semantics and do not
|
||||
mutate observation or linkset collections.
|
||||
|
||||
---
|
||||
|
||||
## 6. Validation & testing
|
||||
|
||||
- **Unit tests** (`StellaOps.Concelier.Core.Tests`) validate schema guards,
|
||||
deterministic linkset hashing, conflict detection fixtures, and supersedes
|
||||
chains.
|
||||
- **Mongo integration tests** (`StellaOps.Concelier.Storage.Mongo.Tests`) verify
|
||||
indexes and idempotent writes under concurrency.
|
||||
- **CLI smoke suites** confirm `stella advisories observations` and `stella
|
||||
advisories linksets` export stable JSON.
|
||||
- **Determinism checks** replay identical upstream payloads and assert that the
|
||||
resulting observation and linkset documents match byte for byte.
|
||||
- **Offline kit verification** simulates air-gapped bootstrap to confirm that
|
||||
snapshots align with live data.
|
||||
|
||||
Add fixtures whenever a new conflict type or correlation signal is introduced.
|
||||
Ensure canonical JSON serialization remains stable across .NET runtime updates.
|
||||
|
||||
---
|
||||
|
||||
## 7. Reviewer checklist
|
||||
|
||||
- Observation schema segment matches the latest `StellaOps.Concelier.Models`
|
||||
contract.
|
||||
- Linkset lifecycle covers correlation signals, conflict classes, and
|
||||
deterministic IDs.
|
||||
- AOC invariants are explicitly called out with violation codes.
|
||||
- Examples include multi-source correlation plus conflict annotation.
|
||||
- Downstream consumer guidance reflects active APIs and CLI features.
|
||||
- Testing section lists required suites (Core, Storage, CLI, Offline).
|
||||
- Imposed rule reminder is present at the top of the document.
|
||||
|
||||
Confirmed against Concelier Link-Not-Merge tasks:
|
||||
`CONCELIER-LNM-21-001..005`, `CONCELIER-LNM-21-101..103`,
|
||||
`CONCELIER-LNM-21-201..203`.
|
||||
|
||||
@@ -1,429 +1,429 @@
|
||||
> **Imposed rule:** Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.
|
||||
|
||||
---
|
||||
|
||||
# Epic 16: Air‑Gapped Mode
|
||||
|
||||
**Short name:** Air‑Gapped Mode
|
||||
**Primary components:** Web Services API, Console, CLI, Orchestrator, Task Runner, Conseiller (Feedser), Excitator (VEXer), Policy Engine, Findings Ledger, Export Center, Authority & Tenancy, Notifications, Observability & Forensics
|
||||
**Surfaces:** offline bootstrap, update ingestion via mirror bundles, sealed egress, deterministic jobs, offline advisories/VEX, offline policy packs, offline notifications, evidence exports
|
||||
**Dependencies:** Export Center, Containerized Distribution, Authority‑Backed Scopes & Tenancy, Observability & Forensics, Policy Studio
|
||||
|
||||
**AOC ground rule reminder:** Conseiller and Excitator aggregate and link advisories/VEX. They never merge or mutate source records. Air‑Gapped Mode must preserve this invariant even when mirroring and importing updates.
|
||||
|
||||
---
|
||||
|
||||
## 1) What it is
|
||||
|
||||
A fully supported operating profile where StellaOps runs in a disconnected environment with:
|
||||
|
||||
* **Zero external egress** from platform services and jobs.
|
||||
* **Deterministic inputs** provided via signed, offline **Mirror Bundles** (advisories, VEX, policy packs, vendor feeds, Stella metadata, container images, dashboards).
|
||||
* **Offline bootstrap** for images and charts, plus reproducible configuration and cryptographically verifiable updates.
|
||||
* **Graceful feature degradation** with explicit UX: features that require external connectivity are either backed by local artifacts or clearly disabled with an explanation.
|
||||
* **Auditable import/export** including provenance attestations, evidence bundles, and chain‑of‑custody for all offline exchanges.
|
||||
|
||||
Air‑Gapped Mode is selectable at install time and enforceable at runtime. When enabled, all components operate under an “egress sealed” policy and only consume data from local stores.
|
||||
|
||||
---
|
||||
|
||||
## 2) Why
|
||||
|
||||
Many users operate in classified, regulated, or high‑sensitivity networks where egress is prohibited. They still need SBOM analysis, policy evaluation, advisory/VEX mapping, and reporting. Air‑Gapped Mode provides the same core outcomes with verifiable offline inputs and explicit operational guardrails.
|
||||
|
||||
---
|
||||
|
||||
## 3) How it should work
|
||||
|
||||
### 3.1 Modes and lifecycle
|
||||
|
||||
* **Connected Mode:** normal operation; can create Mirror Bundles on a staging host.
|
||||
* **Sealed Air‑Gapped Mode:** platform enforces no egress. Only local resources are allowed.
|
||||
* **Transition flow:**
|
||||
|
||||
1. Prepare an offline **Bootstrap Pack** with all container images, Helm/compose charts, seed database, and initial Mirror Bundle.
|
||||
2. Install in the air‑gapped enclave and **seal** egress.
|
||||
3. Periodically import new **Mirror Bundles** via removable media.
|
||||
4. Export evidence/reports as needed.
|
||||
|
||||
### 3.2 Egress sealing
|
||||
|
||||
* **Static guardrails:**
|
||||
|
||||
* Platform flag `STELLA_AIRGAP=sealed` and database feature flag `env.mode='sealed'`.
|
||||
* NetworkPolicy/iptables/eBPF deny‑all egress for namespaces/pods except loopback and the internal object store.
|
||||
* Outbound DNS blocked.
|
||||
* HTTP clients in code use a single `EgressPolicy` facade. When sealed, it panics on direct network calls and returns a typed error with remediation (“import a Mirror Bundle”).
|
||||
* **Verification:** `GET /system/airgap/status` returns `sealed: true|false`, current policy hash, and last import timestamp. CLI prints warning if not sealed in declared air‑gapped install.
|
||||
|
||||
### 3.3 Trusted time
|
||||
|
||||
* Air‑gapped systems cannot NTP. Each Mirror Bundle includes a **signed time token** (Roughtime‑style or RFC 3161) from a trusted authority. On import, platform stores `time_anchor` for drift calculations and staleness checks.
|
||||
* If time drift exceeds policy threshold, UI shows “stale view” badges and some jobs are blocked until a new bundle provides a fresh anchor.
|
||||
|
||||
### 3.4 Mirror Bundles (offline updates)
|
||||
|
||||
* **Content types:**
|
||||
|
||||
* Public advisories (OSV, GHSA, vendor advisories), NVD mappings, CPE/Package metadata.
|
||||
* VEX statements from vendors/communities.
|
||||
* Policy packs (templates, baselines, versioned rule sets).
|
||||
* StellaOps engine metadata and schema migrations.
|
||||
* Optional: **OCI image set** for platform and recommended runners.
|
||||
* Optional: dashboards and alert rule packs.
|
||||
* **Format:** a TUF‑like layout:
|
||||
|
||||
```
|
||||
root.json, snapshot.json, timestamp.json, targets/
|
||||
advisories/*.jsonl.zst
|
||||
vex/*.jsonl.zst
|
||||
policy/*.tar.zst
|
||||
images/* (OCI layout or oci-archive)
|
||||
meta/engine/*.tgz
|
||||
meta/time-anchor.json (signed)
|
||||
```
|
||||
* **Integrity & trust:**
|
||||
|
||||
* DSSE‑signed target manifests.
|
||||
* Root of trust rotated via `root.json` within strict policy; rotation requires manual dual approval in sealed mode.
|
||||
* Each content artifact has a content digest and a **Merkle root** for the overall bundle.
|
||||
* **Creation:** in connected networks, `stella mirror create --content advisories,vex,policy,images --since 2025-01-01 --out bundle.tgz`.
|
||||
* **Import:** in air‑gap, `stella airgap import bundle.tgz`. The importer verifies DSSE, TUF metadata, Merkle root, then writes to local object store and updates catalog tables.
|
||||
* **Idempotence:** imports are content‑addressed; re‑imports deduplicate.
|
||||
|
||||
### 3.5 Deterministic jobs and sources
|
||||
|
||||
* **Allowed sources:** filesystem, internal object store, tenant private registry, and pre‑approved connectors that don’t require external egress.
|
||||
* **Disallowed in sealed mode:** remote package registries, web scrapers, outbound webhooks, cloud KMS unless on the enclave network.
|
||||
* **Runner policy:** the Task Runner verifies job descriptors contain no network calls unless marked `internal:` with allow‑listed destinations. Violations fail at plan time with an explainable error.
|
||||
|
||||
### 3.6 Conseiller and Excitator in air‑gap
|
||||
|
||||
* **Conseiller (Feedser):** ingests advisories only from imported bundles or tenant local feeds. It preserves source identities and never merges. Linkage uses bundle‑provided cross‑refs and local heuristics.
|
||||
* **Excitator (VEXer):** imports VEX records as‑is, links them to components and advisories, and records the origin bundle and statement digests. Consensus Lens (Epic 7) operates offline across the imported sources.
|
||||
|
||||
### 3.7 Policy Engine and Studio
|
||||
|
||||
* Policy packs are versioned and imported via bundles.
|
||||
* Simulation and authoring work locally. Exports of new or updated policies can be packaged as **Policy Sub‑Bundles** for transfer back to connected environments if needed.
|
||||
* Engine shows which rules depend on external evidence and how they degrade in sealed mode (e.g., “No external EPSS; using cached percentile from last bundle.”).
|
||||
|
||||
### 3.8 Notifications in sealed mode
|
||||
|
||||
* Default to **local delivery** only: SMTP relay inside enclave, syslog, file sink.
|
||||
* External webhooks are disabled.
|
||||
* Notification templates show “air‑gap compliant channel” tags to avoid misconfiguration.
|
||||
|
||||
### 3.9 Observability & Forensics
|
||||
|
||||
* Traces, logs, metrics remain local.
|
||||
* Evidence Locker supports **portable evidence packages** for cross‑domain transfer: `stella forensic snapshot create --portable`.
|
||||
* Importing an evidence bundle in another enclave verifies signatures and maintains chain‑of‑custody.
|
||||
|
||||
### 3.10 Console and CLI behavior
|
||||
|
||||
* Console shows a prominent **Air‑Gapped: Sealed** badge with last import time and staleness indicators for advisories, VEX, and policy packs.
|
||||
* CLI commands gain `--sealed` awareness: any operation that would egress prints a refusal with remediation suggesting the appropriate import.
|
||||
|
||||
### 3.11 Multi‑tenant and scope
|
||||
|
||||
* Tenancy works unchanged. Bundle imports can target:
|
||||
|
||||
* `--tenant-global`: shared catalogs (advisories, VEX, policy baselines).
|
||||
* `--tenant=<id>`: tenant‑specific content (e.g., private advisories).
|
||||
* Authority scopes gain `airgap:import`, `airgap:status:read`, `airgap:seal` (admin‑only).
|
||||
|
||||
### 3.12 Feature degradation matrix
|
||||
|
||||
* **AI Assistant:** offline variants use local models if installed; otherwise feature is disabled with a message.
|
||||
* **External reputation feeds (e.g., EPSS‑like):** replaced by cached values from the bundle.
|
||||
* **Container base image lookups:** rely on imported metadata or tenant private registry.
|
||||
|
||||
---
|
||||
|
||||
## 4) Architecture
|
||||
|
||||
### 4.1 New modules
|
||||
|
||||
* `airgap/controller`
|
||||
|
||||
* Sealing state machine; status API; guardrails wiring into HTTP clients and runner.
|
||||
* `airgap/importer`
|
||||
|
||||
* TUF/DSSE verification, Merkle validation, object store loader, catalog updater.
|
||||
* `mirror/creator`
|
||||
|
||||
* Connected‑side builder for bundles; content plug‑ins for advisories/VEX/policy/images.
|
||||
* `airgap/policy`
|
||||
|
||||
* Enforcement library exposing `EgressPolicy` facade and job plan validators.
|
||||
* `airgap/time`
|
||||
|
||||
* Time anchor parser, drift checks, staleness annotations.
|
||||
* `console/airgap`
|
||||
|
||||
* Sealed badge, import UI, staleness dashboards, degradation notices.
|
||||
* `cli/airgap`
|
||||
|
||||
* `stella airgap seal|status|import|verify` commands; `stella mirror create|verify`.
|
||||
|
||||
### 4.2 Data model additions
|
||||
|
||||
* `airgap_state(id, sealed BOOLEAN, policy_hash TEXT, last_import_at TIMESTAMP, time_anchor JSONB)`
|
||||
* `bundle_catalog(id, kind ENUM, merkle_root TEXT, dsse_signer TEXT, created_at TIMESTAMP, imported_at TIMESTAMP, scope ENUM('global','tenant'), tenant_id NULLABLE, labels JSONB)`
|
||||
* `bundle_items(bundle_id, path TEXT, sha256 TEXT, size BIGINT, type TEXT, meta JSONB)`
|
||||
* `import_audit(id, bundle_id, actor, tenant_scope, verify_result, trace_id, created_at)`
|
||||
|
||||
RLS: tenant‑scoped rows when `scope='tenant'`; global rows readable only with `stella:airgap:status:read`.
|
||||
|
||||
### 4.3 Storage layout
|
||||
|
||||
Object store paths:
|
||||
|
||||
```
|
||||
tenants/_global/mirror/<bundle_id>/targets/...
|
||||
tenants/<tenant>/mirror/<bundle_id>/targets/...
|
||||
tenants/_global/images/<digest>/...
|
||||
```
|
||||
|
||||
Evidence locker remains separate. Imported images use **OCI layout** for local registry sync.
|
||||
|
||||
### 4.4 Message topics
|
||||
|
||||
* `stella.<tenant>.airgap.imported` with bundle metadata.
|
||||
* `stella.<tenant>.airgap.staleness` periodic events emitted for UX.
|
||||
* `stella.<tenant>.policy.degraded` when rules fall back due to sealed mode.
|
||||
|
||||
---
|
||||
|
||||
## 5) APIs and contracts
|
||||
|
||||
### 5.1 Status and control
|
||||
|
||||
* `GET /system/airgap/status` → `{ sealed, policy_hash, last_import_at, time_anchor, drift_seconds, staleness: { advisories_days, vex_days, policy_days } }`
|
||||
* `POST /system/airgap/seal` → seals environment; requires `stella:airgap:seal#tenant/<id or global>`.
|
||||
* `POST /system/airgap/unseal` → only allowed if installed mode is not declared “permanently sealed” at bootstrap. Typically disabled.
|
||||
|
||||
### 5.2 Import & verify
|
||||
|
||||
* `POST /airgap/import` multipart or file reference → runs verify, writes catalog, returns bundle summary and warnings.
|
||||
* `POST /airgap/verify` dry‑run verification returning DSSE/TUF and Merkle results.
|
||||
* `GET /airgap/bundles` list imported bundles with filters.
|
||||
|
||||
### 5.3 Conseiller/Excitator sources
|
||||
|
||||
* `POST /feeds/register` supports `kind=mirror` with `bundle_id` and paths; disallowed to point to external URLs in sealed mode.
|
||||
* `GET /feeds/status` shows per‑source staleness and last artifact version.
|
||||
|
||||
### 5.4 Errors
|
||||
|
||||
Standardized sealed‑mode error:
|
||||
|
||||
```
|
||||
{
|
||||
"code": "AIRGAP_EGRESS_BLOCKED",
|
||||
"message": "Egress is sealed. Import a Mirror Bundle with advisories.",
|
||||
"remediation": "Run: stella airgap import bundle.tgz",
|
||||
"trace_id": "..."
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 6) Documentation changes
|
||||
|
||||
Create or update:
|
||||
|
||||
1. `/docs/airgap/overview.md`
|
||||
|
||||
* Modes, lifecycle, responsibilities, threat model, what degrades.
|
||||
2. `/docs/airgap/bootstrap.md`
|
||||
|
||||
* Offline Bootstrap Pack creation, validation, install steps for Helm/compose, local registry seeding.
|
||||
3. `/docs/airgap/mirror-bundles.md`
|
||||
|
||||
* Bundle format, DSSE/TUF/Merkle, signed time, creation on connected host, import in sealed environment, rotation of roots.
|
||||
4. `/docs/airgap/sealing-and-egress.md`
|
||||
|
||||
* Network policies, EgressPolicy facade, runner validation, verifying sealed status.
|
||||
5. `/docs/airgap/staleness-and-time.md`
|
||||
|
||||
* Time anchor, drift, staleness budgets and UI behavior.
|
||||
6. `/docs/airgap/operations.md`
|
||||
|
||||
* Periodic update cadence, runbooks, failure scenarios, disaster recovery.
|
||||
7. `/docs/airgap/degradation-matrix.md`
|
||||
|
||||
* Feature map: available, degraded, disabled; with remediation.
|
||||
8. `/docs/console/airgap.md`
|
||||
|
||||
* Status badges, import wizard, staleness indicators.
|
||||
9. `/docs/cli/airgap.md`
|
||||
|
||||
* Commands, examples, exit codes.
|
||||
10. `/docs/security/trust-and-signing.md`
|
||||
|
||||
* Roots of trust, key rotation, DSSE, TUF model.
|
||||
|
||||
11. `/docs/dev/airgap-contracts.md`
|
||||
|
||||
* EgressPolicy usage, testing patterns, sealed‑mode CI gates.
|
||||
|
||||
Add the banner at the top of each page:
|
||||
|
||||
> **Imposed rule:** Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.
|
||||
|
||||
---
|
||||
|
||||
## 7) Implementation plan
|
||||
|
||||
### Phase 1 — Foundations
|
||||
|
||||
* Add `airgap/controller` with sealed state and status API.
|
||||
* Integrate `EgressPolicy` facade in all outbound network call sites.
|
||||
* Provide default NetworkPolicy/iptables templates and Helm values to block egress.
|
||||
* Console shows sealed badge and status.
|
||||
|
||||
### Phase 2 — Mirror Bundles
|
||||
|
||||
* Implement `mirror/creator` in connected mode with content plug‑ins.
|
||||
* Implement `airgap/importer` with DSSE/TUF/Merkle verification and catalog updates.
|
||||
* Export Center gains **Mirror bundle** build and verify commands (connected side).
|
||||
|
||||
### Phase 3 — Deterministic jobs
|
||||
|
||||
* Add job plan validation in the Task Runner.
|
||||
* Restrict sources in sealed mode.
|
||||
* Conseiller/Excitator add “mirror source” adapters.
|
||||
|
||||
### Phase 4 — Staleness and time
|
||||
|
||||
* Parse time anchors; enforce staleness budgets; add UI indicators and task refusal when budgets exceeded.
|
||||
* Notifications for expiring anchors.
|
||||
|
||||
### Phase 5 — Degradation matrix and UX
|
||||
|
||||
* Wire feature flags and fallbacks in Console and APIs.
|
||||
* Improve error messages with remediation guidance.
|
||||
|
||||
### Phase 6 — Evidence portability
|
||||
|
||||
* Portable evidence packages: export/import with full verification.
|
||||
* Document cross‑domain workflows.
|
||||
|
||||
---
|
||||
|
||||
## 8) Engineering tasks
|
||||
|
||||
**Air‑gap controller and sealing**
|
||||
|
||||
* [ ] Implement `airgap/controller` with persistent state and RBAC.
|
||||
* [ ] Add `GET /system/airgap/status`, `POST /system/airgap/seal`.
|
||||
* [ ] Provide cluster egress templates for Kubernetes and for docker‑compose.
|
||||
* [ ] Instrument startup checks to refuse running in sealed mode if egress rules aren’t applied.
|
||||
|
||||
**EgressPolicy integration**
|
||||
|
||||
* [ ] Create `pkg/egress` facade and replace all direct HTTP client constructions in services.
|
||||
* [ ] Add linter rule and CI check forbidding raw `http.NewClient` in server code.
|
||||
* [ ] Add unit tests for sealed and unsealed behavior.
|
||||
|
||||
**Mirror bundles**
|
||||
|
||||
* [ ] Implement TUF/DSSE verifiers and Merkle root builder.
|
||||
* [ ] Build content plug‑ins: advisories, VEX, policy packs, images.
|
||||
* [ ] Write `bundle_catalog` and `bundle_items` tables with RLS.
|
||||
* [ ] CLI: `stella mirror create|verify`, `stella airgap import|verify`.
|
||||
|
||||
**Conseiller/Excitator**
|
||||
|
||||
* [ ] Add mirror adapters for read‑only ingestion from bundle paths.
|
||||
* [ ] Persist source digests and bundle IDs on each linked record.
|
||||
* [ ] Unit tests to ensure no merge behavior is introduced by bundle ingestion.
|
||||
|
||||
**Policy Engine & Studio**
|
||||
|
||||
* [ ] Accept policy packs from bundles; track `policy_version` and `bundle_id`.
|
||||
* [ ] Add degradation notices for rules requiring external reputation; provide cached fallbacks.
|
||||
|
||||
**Task Runner & Orchestrator**
|
||||
|
||||
* [ ] Plan‑time validation against network calls; add `internal:` allow‑list mapping.
|
||||
* [ ] Emit sealed‑mode violations to Timeline with remediation text.
|
||||
|
||||
**Console**
|
||||
|
||||
* [ ] Status panel: sealed badge, last import, staleness meters.
|
||||
* [ ] Import wizard with verify results and catalog diff preview.
|
||||
* [ ] Degradation matrix UI and contextual tooltips.
|
||||
|
||||
**Observability & Forensics**
|
||||
|
||||
* [ ] Mark sealed mode in telemetry attributes.
|
||||
* [ ] Add portable evidence package export/import; verify on read.
|
||||
|
||||
**Authority & Tenancy**
|
||||
|
||||
* [ ] New scopes: `airgap:seal`, `airgap:import`, `airgap:status:read`.
|
||||
* [ ] Audit import actions with actor and trace ID.
|
||||
|
||||
**Docs**
|
||||
|
||||
* [ ] Author all pages listed in section 6, include signed‑time workflow diagrams.
|
||||
* [ ] Insert banner statement in each page.
|
||||
|
||||
**Testing**
|
||||
|
||||
* [ ] Sealed‑mode e2e: attempt egress; ensure refusal and remediation.
|
||||
* [ ] Bundle import e2e: corrupt DSSE, wrong root, tampered artifact → rejected.
|
||||
* [ ] Performance: large advisory bundle import within target time (see Acceptance).
|
||||
* [ ] Time drift scenarios and staleness budget enforcement.
|
||||
* [ ] Regression: ensure AOC rules unchanged in sealed mode.
|
||||
|
||||
---
|
||||
|
||||
## 9) Feature changes required in other components
|
||||
|
||||
* **Export Center:** add mirror bundle export profile, signed‑time token inclusion, and portable evidence packages.
|
||||
* **Notifications:** remove external webhooks by default in sealed mode; add local SMTP/syslog sinks.
|
||||
* **CLI Parity:** ensure all admin and import operations are exposed; add sealed‑mode safety prompts.
|
||||
* **Containerized Distribution:** ship **Bootstrap Pack** that includes all images and charts in a single oci‑archive set with index manifest.
|
||||
* **Observability:** disable remote exporters; include local dashboards; mark sealed mode in UI.
|
||||
* **Policy Studio:** enable offline authoring and export of policy sub‑bundles.
|
||||
* **VEX Consensus Lens:** ensure it operates solely on imported VEX statements; highlight coverage vs. stale.
|
||||
|
||||
> **Imposed rule reminder:** Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.
|
||||
|
||||
---
|
||||
|
||||
## 10) Acceptance criteria
|
||||
|
||||
* Environment can be **sealed** and verified via API, CLI, and network policies.
|
||||
* Import of a valid Mirror Bundle succeeds; DSSE, TUF, and Merkle validations recorded in `import_audit`.
|
||||
* Conseiller and Excitator operate only on imported sources; linkage reflects original source identities.
|
||||
* Policy packs are importable and versioned; rules that depend on external evidence show clear degradation.
|
||||
* Large bundle (e.g., 8–12 GB with images) imports in under 20 minutes on SSD storage and indexes advisories in under 5 minutes on a 4‑core node.
|
||||
* Console displays sealed badge, last import, staleness, and degradation matrix.
|
||||
* Attempted egress in sealed mode fails with `AIRGAP_EGRESS_BLOCKED` and remediation.
|
||||
* Portable evidence packages export and verify across separate enclaves.
|
||||
* All changes documented with the banner statement.
|
||||
|
||||
---
|
||||
|
||||
## 11) Risks and mitigations
|
||||
|
||||
* **Key management complexity:** rotate TUF roots with dual‑control workflow and explicit docs; fail‑safe to previous root if rotation bundle absent.
|
||||
* **Staleness risk:** enforce budgets and block risk‑critical jobs when expired; provide monitoring and notifications for impending staleness.
|
||||
* **Operator error during import:** dry‑run verification, diff preview of catalog changes, and ability to roll back via content address.
|
||||
* **Hidden egress paths:** CI lints and runtime guardrails; network policies enforced at cluster layer.
|
||||
* **Bundle size bloat:** Zstandard compression, delta bundles, and selective content flags for creation.
|
||||
|
||||
---
|
||||
|
||||
## 12) Philosophy
|
||||
|
||||
* **Predictable over perfect:** deterministic, explainable results beat unknown “live” results in sensitive networks.
|
||||
* **Trust is earned:** every offline exchange is signed, verifiable, and auditable.
|
||||
* **Degrade transparently:** when features reduce capability, explain it and guide remediation.
|
||||
|
||||
> **Imposed rule:** Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.
|
||||
> **Imposed rule:** Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.
|
||||
|
||||
---
|
||||
|
||||
# Epic 16: Air‑Gapped Mode
|
||||
|
||||
**Short name:** Air‑Gapped Mode
|
||||
**Primary components:** Web Services API, Console, CLI, Orchestrator, Task Runner, Conseiller (Feedser), Excitator (VEXer), Policy Engine, Findings Ledger, Export Center, Authority & Tenancy, Notifications, Observability & Forensics
|
||||
**Surfaces:** offline bootstrap, update ingestion via mirror bundles, sealed egress, deterministic jobs, offline advisories/VEX, offline policy packs, offline notifications, evidence exports
|
||||
**Dependencies:** Export Center, Containerized Distribution, Authority‑Backed Scopes & Tenancy, Observability & Forensics, Policy Studio
|
||||
|
||||
**AOC ground rule reminder:** Conseiller and Excitator aggregate and link advisories/VEX. They never merge or mutate source records. Air‑Gapped Mode must preserve this invariant even when mirroring and importing updates.
|
||||
|
||||
---
|
||||
|
||||
## 1) What it is
|
||||
|
||||
A fully supported operating profile where StellaOps runs in a disconnected environment with:
|
||||
|
||||
* **Zero external egress** from platform services and jobs.
|
||||
* **Deterministic inputs** provided via signed, offline **Mirror Bundles** (advisories, VEX, policy packs, vendor feeds, Stella metadata, container images, dashboards).
|
||||
* **Offline bootstrap** for images and charts, plus reproducible configuration and cryptographically verifiable updates.
|
||||
* **Graceful feature degradation** with explicit UX: features that require external connectivity are either backed by local artifacts or clearly disabled with an explanation.
|
||||
* **Auditable import/export** including provenance attestations, evidence bundles, and chain‑of‑custody for all offline exchanges.
|
||||
|
||||
Air‑Gapped Mode is selectable at install time and enforceable at runtime. When enabled, all components operate under an “egress sealed” policy and only consume data from local stores.
|
||||
|
||||
---
|
||||
|
||||
## 2) Why
|
||||
|
||||
Many users operate in classified, regulated, or high‑sensitivity networks where egress is prohibited. They still need SBOM analysis, policy evaluation, advisory/VEX mapping, and reporting. Air‑Gapped Mode provides the same core outcomes with verifiable offline inputs and explicit operational guardrails.
|
||||
|
||||
---
|
||||
|
||||
## 3) How it should work
|
||||
|
||||
### 3.1 Modes and lifecycle
|
||||
|
||||
* **Connected Mode:** normal operation; can create Mirror Bundles on a staging host.
|
||||
* **Sealed Air‑Gapped Mode:** platform enforces no egress. Only local resources are allowed.
|
||||
* **Transition flow:**
|
||||
|
||||
1. Prepare an offline **Bootstrap Pack** with all container images, Helm/compose charts, seed database, and initial Mirror Bundle.
|
||||
2. Install in the air‑gapped enclave and **seal** egress.
|
||||
3. Periodically import new **Mirror Bundles** via removable media.
|
||||
4. Export evidence/reports as needed.
|
||||
|
||||
### 3.2 Egress sealing
|
||||
|
||||
* **Static guardrails:**
|
||||
|
||||
* Platform flag `STELLA_AIRGAP=sealed` and database feature flag `env.mode='sealed'`.
|
||||
* NetworkPolicy/iptables/eBPF deny‑all egress for namespaces/pods except loopback and the internal object store.
|
||||
* Outbound DNS blocked.
|
||||
* HTTP clients in code use a single `EgressPolicy` facade. When sealed, it panics on direct network calls and returns a typed error with remediation (“import a Mirror Bundle”).
|
||||
* **Verification:** `GET /system/airgap/status` returns `sealed: true|false`, current policy hash, and last import timestamp. CLI prints warning if not sealed in declared air‑gapped install.
|
||||
|
||||
### 3.3 Trusted time
|
||||
|
||||
* Air‑gapped systems cannot NTP. Each Mirror Bundle includes a **signed time token** (Roughtime‑style or RFC 3161) from a trusted authority. On import, platform stores `time_anchor` for drift calculations and staleness checks.
|
||||
* If time drift exceeds policy threshold, UI shows “stale view” badges and some jobs are blocked until a new bundle provides a fresh anchor.
|
||||
|
||||
### 3.4 Mirror Bundles (offline updates)
|
||||
|
||||
* **Content types:**
|
||||
|
||||
* Public advisories (OSV, GHSA, vendor advisories), NVD mappings, CPE/Package metadata.
|
||||
* VEX statements from vendors/communities.
|
||||
* Policy packs (templates, baselines, versioned rule sets).
|
||||
* StellaOps engine metadata and schema migrations.
|
||||
* Optional: **OCI image set** for platform and recommended runners.
|
||||
* Optional: dashboards and alert rule packs.
|
||||
* **Format:** a TUF‑like layout:
|
||||
|
||||
```
|
||||
root.json, snapshot.json, timestamp.json, targets/
|
||||
advisories/*.jsonl.zst
|
||||
vex/*.jsonl.zst
|
||||
policy/*.tar.zst
|
||||
images/* (OCI layout or oci-archive)
|
||||
meta/engine/*.tgz
|
||||
meta/time-anchor.json (signed)
|
||||
```
|
||||
* **Integrity & trust:**
|
||||
|
||||
* DSSE‑signed target manifests.
|
||||
* Root of trust rotated via `root.json` within strict policy; rotation requires manual dual approval in sealed mode.
|
||||
* Each content artifact has a content digest and a **Merkle root** for the overall bundle.
|
||||
* **Creation:** in connected networks, `stella mirror create --content advisories,vex,policy,images --since 2025-01-01 --out bundle.tgz`.
|
||||
* **Import:** in air‑gap, `stella airgap import bundle.tgz`. The importer verifies DSSE, TUF metadata, Merkle root, then writes to local object store and updates catalog tables.
|
||||
* **Idempotence:** imports are content‑addressed; re‑imports deduplicate.
|
||||
|
||||
### 3.5 Deterministic jobs and sources
|
||||
|
||||
* **Allowed sources:** filesystem, internal object store, tenant private registry, and pre‑approved connectors that don’t require external egress.
|
||||
* **Disallowed in sealed mode:** remote package registries, web scrapers, outbound webhooks, cloud KMS unless on the enclave network.
|
||||
* **Runner policy:** the Task Runner verifies job descriptors contain no network calls unless marked `internal:` with allow‑listed destinations. Violations fail at plan time with an explainable error.
|
||||
|
||||
### 3.6 Conseiller and Excitator in air‑gap
|
||||
|
||||
* **Conseiller (Feedser):** ingests advisories only from imported bundles or tenant local feeds. It preserves source identities and never merges. Linkage uses bundle‑provided cross‑refs and local heuristics.
|
||||
* **Excitator (VEXer):** imports VEX records as‑is, links them to components and advisories, and records the origin bundle and statement digests. Consensus Lens (Epic 7) operates offline across the imported sources.
|
||||
|
||||
### 3.7 Policy Engine and Studio
|
||||
|
||||
* Policy packs are versioned and imported via bundles.
|
||||
* Simulation and authoring work locally. Exports of new or updated policies can be packaged as **Policy Sub‑Bundles** for transfer back to connected environments if needed.
|
||||
* Engine shows which rules depend on external evidence and how they degrade in sealed mode (e.g., “No external EPSS; using cached percentile from last bundle.”).
|
||||
|
||||
### 3.8 Notifications in sealed mode
|
||||
|
||||
* Default to **local delivery** only: SMTP relay inside enclave, syslog, file sink.
|
||||
* External webhooks are disabled.
|
||||
* Notification templates show “air‑gap compliant channel” tags to avoid misconfiguration.
|
||||
|
||||
### 3.9 Observability & Forensics
|
||||
|
||||
* Traces, logs, metrics remain local.
|
||||
* Evidence Locker supports **portable evidence packages** for cross‑domain transfer: `stella forensic snapshot create --portable`.
|
||||
* Importing an evidence bundle in another enclave verifies signatures and maintains chain‑of‑custody.
|
||||
|
||||
### 3.10 Console and CLI behavior
|
||||
|
||||
* Console shows a prominent **Air‑Gapped: Sealed** badge with last import time and staleness indicators for advisories, VEX, and policy packs.
|
||||
* CLI commands gain `--sealed` awareness: any operation that would egress prints a refusal with remediation suggesting the appropriate import.
|
||||
|
||||
### 3.11 Multi‑tenant and scope
|
||||
|
||||
* Tenancy works unchanged. Bundle imports can target:
|
||||
|
||||
* `--tenant-global`: shared catalogs (advisories, VEX, policy baselines).
|
||||
* `--tenant=<id>`: tenant‑specific content (e.g., private advisories).
|
||||
* Authority scopes gain `airgap:import`, `airgap:status:read`, `airgap:seal` (admin‑only).
|
||||
|
||||
### 3.12 Feature degradation matrix
|
||||
|
||||
* **AI Assistant:** offline variants use local models if installed; otherwise feature is disabled with a message.
|
||||
* **External reputation feeds (e.g., EPSS‑like):** replaced by cached values from the bundle.
|
||||
* **Container base image lookups:** rely on imported metadata or tenant private registry.
|
||||
|
||||
---
|
||||
|
||||
## 4) Architecture
|
||||
|
||||
### 4.1 New modules
|
||||
|
||||
* `airgap/controller`
|
||||
|
||||
* Sealing state machine; status API; guardrails wiring into HTTP clients and runner.
|
||||
* `airgap/importer`
|
||||
|
||||
* TUF/DSSE verification, Merkle validation, object store loader, catalog updater.
|
||||
* `mirror/creator`
|
||||
|
||||
* Connected‑side builder for bundles; content plug‑ins for advisories/VEX/policy/images.
|
||||
* `airgap/policy`
|
||||
|
||||
* Enforcement library exposing `EgressPolicy` facade and job plan validators.
|
||||
* `airgap/time`
|
||||
|
||||
* Time anchor parser, drift checks, staleness annotations.
|
||||
* `console/airgap`
|
||||
|
||||
* Sealed badge, import UI, staleness dashboards, degradation notices.
|
||||
* `cli/airgap`
|
||||
|
||||
* `stella airgap seal|status|import|verify` commands; `stella mirror create|verify`.
|
||||
|
||||
### 4.2 Data model additions
|
||||
|
||||
* `airgap_state(id, sealed BOOLEAN, policy_hash TEXT, last_import_at TIMESTAMP, time_anchor JSONB)`
|
||||
* `bundle_catalog(id, kind ENUM, merkle_root TEXT, dsse_signer TEXT, created_at TIMESTAMP, imported_at TIMESTAMP, scope ENUM('global','tenant'), tenant_id NULLABLE, labels JSONB)`
|
||||
* `bundle_items(bundle_id, path TEXT, sha256 TEXT, size BIGINT, type TEXT, meta JSONB)`
|
||||
* `import_audit(id, bundle_id, actor, tenant_scope, verify_result, trace_id, created_at)`
|
||||
|
||||
RLS: tenant‑scoped rows when `scope='tenant'`; global rows readable only with `stella:airgap:status:read`.
|
||||
|
||||
### 4.3 Storage layout
|
||||
|
||||
Object store paths:
|
||||
|
||||
```
|
||||
tenants/_global/mirror/<bundle_id>/targets/...
|
||||
tenants/<tenant>/mirror/<bundle_id>/targets/...
|
||||
tenants/_global/images/<digest>/...
|
||||
```
|
||||
|
||||
Evidence locker remains separate. Imported images use **OCI layout** for local registry sync.
|
||||
|
||||
### 4.4 Message topics
|
||||
|
||||
* `stella.<tenant>.airgap.imported` with bundle metadata.
|
||||
* `stella.<tenant>.airgap.staleness` periodic events emitted for UX.
|
||||
* `stella.<tenant>.policy.degraded` when rules fall back due to sealed mode.
|
||||
|
||||
---
|
||||
|
||||
## 5) APIs and contracts
|
||||
|
||||
### 5.1 Status and control
|
||||
|
||||
* `GET /system/airgap/status` → `{ sealed, policy_hash, last_import_at, time_anchor, drift_seconds, staleness: { advisories_days, vex_days, policy_days } }`
|
||||
* `POST /system/airgap/seal` → seals environment; requires `stella:airgap:seal#tenant/<id or global>`.
|
||||
* `POST /system/airgap/unseal` → only allowed if installed mode is not declared “permanently sealed” at bootstrap. Typically disabled.
|
||||
|
||||
### 5.2 Import & verify
|
||||
|
||||
* `POST /airgap/import` multipart or file reference → runs verify, writes catalog, returns bundle summary and warnings.
|
||||
* `POST /airgap/verify` dry‑run verification returning DSSE/TUF and Merkle results.
|
||||
* `GET /airgap/bundles` list imported bundles with filters.
|
||||
|
||||
### 5.3 Conseiller/Excitator sources
|
||||
|
||||
* `POST /feeds/register` supports `kind=mirror` with `bundle_id` and paths; disallowed to point to external URLs in sealed mode.
|
||||
* `GET /feeds/status` shows per‑source staleness and last artifact version.
|
||||
|
||||
### 5.4 Errors
|
||||
|
||||
Standardized sealed‑mode error:
|
||||
|
||||
```
|
||||
{
|
||||
"code": "AIRGAP_EGRESS_BLOCKED",
|
||||
"message": "Egress is sealed. Import a Mirror Bundle with advisories.",
|
||||
"remediation": "Run: stella airgap import bundle.tgz",
|
||||
"trace_id": "..."
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 6) Documentation changes
|
||||
|
||||
Create or update:
|
||||
|
||||
1. `/docs/airgap/overview.md`
|
||||
|
||||
* Modes, lifecycle, responsibilities, threat model, what degrades.
|
||||
2. `/docs/airgap/bootstrap.md`
|
||||
|
||||
* Offline Bootstrap Pack creation, validation, install steps for Helm/compose, local registry seeding.
|
||||
3. `/docs/airgap/mirror-bundles.md`
|
||||
|
||||
* Bundle format, DSSE/TUF/Merkle, signed time, creation on connected host, import in sealed environment, rotation of roots.
|
||||
4. `/docs/airgap/sealing-and-egress.md`
|
||||
|
||||
* Network policies, EgressPolicy facade, runner validation, verifying sealed status.
|
||||
5. `/docs/airgap/staleness-and-time.md`
|
||||
|
||||
* Time anchor, drift, staleness budgets and UI behavior.
|
||||
6. `/docs/airgap/operations.md`
|
||||
|
||||
* Periodic update cadence, runbooks, failure scenarios, disaster recovery.
|
||||
7. `/docs/airgap/degradation-matrix.md`
|
||||
|
||||
* Feature map: available, degraded, disabled; with remediation.
|
||||
8. `/docs/console/airgap.md`
|
||||
|
||||
* Status badges, import wizard, staleness indicators.
|
||||
9. `/docs/cli/airgap.md`
|
||||
|
||||
* Commands, examples, exit codes.
|
||||
10. `/docs/security/trust-and-signing.md`
|
||||
|
||||
* Roots of trust, key rotation, DSSE, TUF model.
|
||||
|
||||
11. `/docs/dev/airgap-contracts.md`
|
||||
|
||||
* EgressPolicy usage, testing patterns, sealed‑mode CI gates.
|
||||
|
||||
Add the banner at the top of each page:
|
||||
|
||||
> **Imposed rule:** Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.
|
||||
|
||||
---
|
||||
|
||||
## 7) Implementation plan
|
||||
|
||||
### Phase 1 — Foundations
|
||||
|
||||
* Add `airgap/controller` with sealed state and status API.
|
||||
* Integrate `EgressPolicy` facade in all outbound network call sites.
|
||||
* Provide default NetworkPolicy/iptables templates and Helm values to block egress.
|
||||
* Console shows sealed badge and status.
|
||||
|
||||
### Phase 2 — Mirror Bundles
|
||||
|
||||
* Implement `mirror/creator` in connected mode with content plug‑ins.
|
||||
* Implement `airgap/importer` with DSSE/TUF/Merkle verification and catalog updates.
|
||||
* Export Center gains **Mirror bundle** build and verify commands (connected side).
|
||||
|
||||
### Phase 3 — Deterministic jobs
|
||||
|
||||
* Add job plan validation in the Task Runner.
|
||||
* Restrict sources in sealed mode.
|
||||
* Conseiller/Excitator add “mirror source” adapters.
|
||||
|
||||
### Phase 4 — Staleness and time
|
||||
|
||||
* Parse time anchors; enforce staleness budgets; add UI indicators and task refusal when budgets exceeded.
|
||||
* Notifications for expiring anchors.
|
||||
|
||||
### Phase 5 — Degradation matrix and UX
|
||||
|
||||
* Wire feature flags and fallbacks in Console and APIs.
|
||||
* Improve error messages with remediation guidance.
|
||||
|
||||
### Phase 6 — Evidence portability
|
||||
|
||||
* Portable evidence packages: export/import with full verification.
|
||||
* Document cross‑domain workflows.
|
||||
|
||||
---
|
||||
|
||||
## 8) Engineering tasks
|
||||
|
||||
**Air‑gap controller and sealing**
|
||||
|
||||
* [ ] Implement `airgap/controller` with persistent state and RBAC.
|
||||
* [ ] Add `GET /system/airgap/status`, `POST /system/airgap/seal`.
|
||||
* [ ] Provide cluster egress templates for Kubernetes and for docker‑compose.
|
||||
* [ ] Instrument startup checks to refuse running in sealed mode if egress rules aren’t applied.
|
||||
|
||||
**EgressPolicy integration**
|
||||
|
||||
* [ ] Create `pkg/egress` facade and replace all direct HTTP client constructions in services.
|
||||
* [ ] Add linter rule and CI check forbidding raw `http.NewClient` in server code.
|
||||
* [ ] Add unit tests for sealed and unsealed behavior.
|
||||
|
||||
**Mirror bundles**
|
||||
|
||||
* [ ] Implement TUF/DSSE verifiers and Merkle root builder.
|
||||
* [ ] Build content plug‑ins: advisories, VEX, policy packs, images.
|
||||
* [ ] Write `bundle_catalog` and `bundle_items` tables with RLS.
|
||||
* [ ] CLI: `stella mirror create|verify`, `stella airgap import|verify`.
|
||||
|
||||
**Conseiller/Excitator**
|
||||
|
||||
* [ ] Add mirror adapters for read‑only ingestion from bundle paths.
|
||||
* [ ] Persist source digests and bundle IDs on each linked record.
|
||||
* [ ] Unit tests to ensure no merge behavior is introduced by bundle ingestion.
|
||||
|
||||
**Policy Engine & Studio**
|
||||
|
||||
* [ ] Accept policy packs from bundles; track `policy_version` and `bundle_id`.
|
||||
* [ ] Add degradation notices for rules requiring external reputation; provide cached fallbacks.
|
||||
|
||||
**Task Runner & Orchestrator**
|
||||
|
||||
* [ ] Plan‑time validation against network calls; add `internal:` allow‑list mapping.
|
||||
* [ ] Emit sealed‑mode violations to Timeline with remediation text.
|
||||
|
||||
**Console**
|
||||
|
||||
* [ ] Status panel: sealed badge, last import, staleness meters.
|
||||
* [ ] Import wizard with verify results and catalog diff preview.
|
||||
* [ ] Degradation matrix UI and contextual tooltips.
|
||||
|
||||
**Observability & Forensics**
|
||||
|
||||
* [ ] Mark sealed mode in telemetry attributes.
|
||||
* [ ] Add portable evidence package export/import; verify on read.
|
||||
|
||||
**Authority & Tenancy**
|
||||
|
||||
* [ ] New scopes: `airgap:seal`, `airgap:import`, `airgap:status:read`.
|
||||
* [ ] Audit import actions with actor and trace ID.
|
||||
|
||||
**Docs**
|
||||
|
||||
* [ ] Author all pages listed in section 6, include signed‑time workflow diagrams.
|
||||
* [ ] Insert banner statement in each page.
|
||||
|
||||
**Testing**
|
||||
|
||||
* [ ] Sealed‑mode e2e: attempt egress; ensure refusal and remediation.
|
||||
* [ ] Bundle import e2e: corrupt DSSE, wrong root, tampered artifact → rejected.
|
||||
* [ ] Performance: large advisory bundle import within target time (see Acceptance).
|
||||
* [ ] Time drift scenarios and staleness budget enforcement.
|
||||
* [ ] Regression: ensure AOC rules unchanged in sealed mode.
|
||||
|
||||
---
|
||||
|
||||
## 9) Feature changes required in other components
|
||||
|
||||
* **Export Center:** add mirror bundle export profile, signed‑time token inclusion, and portable evidence packages.
|
||||
* **Notifications:** remove external webhooks by default in sealed mode; add local SMTP/syslog sinks.
|
||||
* **CLI Parity:** ensure all admin and import operations are exposed; add sealed‑mode safety prompts.
|
||||
* **Containerized Distribution:** ship **Bootstrap Pack** that includes all images and charts in a single oci‑archive set with index manifest.
|
||||
* **Observability:** disable remote exporters; include local dashboards; mark sealed mode in UI.
|
||||
* **Policy Studio:** enable offline authoring and export of policy sub‑bundles.
|
||||
* **VEX Consensus Lens:** ensure it operates solely on imported VEX statements; highlight coverage vs. stale.
|
||||
|
||||
> **Imposed rule reminder:** Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.
|
||||
|
||||
---
|
||||
|
||||
## 10) Acceptance criteria
|
||||
|
||||
* Environment can be **sealed** and verified via API, CLI, and network policies.
|
||||
* Import of a valid Mirror Bundle succeeds; DSSE, TUF, and Merkle validations recorded in `import_audit`.
|
||||
* Conseiller and Excitator operate only on imported sources; linkage reflects original source identities.
|
||||
* Policy packs are importable and versioned; rules that depend on external evidence show clear degradation.
|
||||
* Large bundle (e.g., 8–12 GB with images) imports in under 20 minutes on SSD storage and indexes advisories in under 5 minutes on a 4‑core node.
|
||||
* Console displays sealed badge, last import, staleness, and degradation matrix.
|
||||
* Attempted egress in sealed mode fails with `AIRGAP_EGRESS_BLOCKED` and remediation.
|
||||
* Portable evidence packages export and verify across separate enclaves.
|
||||
* All changes documented with the banner statement.
|
||||
|
||||
---
|
||||
|
||||
## 11) Risks and mitigations
|
||||
|
||||
* **Key management complexity:** rotate TUF roots with dual‑control workflow and explicit docs; fail‑safe to previous root if rotation bundle absent.
|
||||
* **Staleness risk:** enforce budgets and block risk‑critical jobs when expired; provide monitoring and notifications for impending staleness.
|
||||
* **Operator error during import:** dry‑run verification, diff preview of catalog changes, and ability to roll back via content address.
|
||||
* **Hidden egress paths:** CI lints and runtime guardrails; network policies enforced at cluster layer.
|
||||
* **Bundle size bloat:** Zstandard compression, delta bundles, and selective content flags for creation.
|
||||
|
||||
---
|
||||
|
||||
## 12) Philosophy
|
||||
|
||||
* **Predictable over perfect:** deterministic, explainable results beat unknown “live” results in sensitive networks.
|
||||
* **Trust is earned:** every offline exchange is signed, verifiable, and auditable.
|
||||
* **Degrade transparently:** when features reduce capability, explain it and guide remediation.
|
||||
|
||||
> **Imposed rule:** Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.
|
||||
|
||||
@@ -1,13 +1,13 @@
|
||||
# Aggregation-Only Contract (AOC) Guardrails
|
||||
|
||||
The Aggregation-Only Contract keeps ingestion services deterministic and policy-neutral. Use these checkpoints whenever you add or modify backlog items:
|
||||
|
||||
1. **Ingestion writes raw facts only.** Concelier and Excititor append immutable observations/linksets. No precedence, severity, suppression, or "safe fix" hints may be computed at ingest time.
|
||||
2. **Derived semantics live elsewhere.** Policy Engine overlays, Vuln Explorer composition, and downstream reporting layers attach severity, precedence, policy verdicts, and UI hints.
|
||||
3. **Provenance is mandatory.** Every ingestion write must include original source metadata, digests, and signing/provenance evidence when available. Reject writes lacking provenance.
|
||||
4. **Deterministic outputs.** Given the same inputs, ingestion must produce identical documents, hashes, and event payloads across reruns.
|
||||
5. **Guardrails everywhere.** Roslyn analyzers, schema validators, and CI smoke tests should fail builds that attempt forbidden writes.
|
||||
|
||||
For detailed roles and ownership boundaries, see `AGENTS.md` at the repo root and the module-specific `ARCHITECTURE_*.md` dossiers.
|
||||
|
||||
Need the full contract? Read the [Aggregation-Only Contract reference](../ingestion/aggregation-only-contract.md) for schemas, error codes, and migration guidance.
|
||||
# Aggregation-Only Contract (AOC) Guardrails
|
||||
|
||||
The Aggregation-Only Contract keeps ingestion services deterministic and policy-neutral. Use these checkpoints whenever you add or modify backlog items:
|
||||
|
||||
1. **Ingestion writes raw facts only.** Concelier and Excititor append immutable observations/linksets. No precedence, severity, suppression, or "safe fix" hints may be computed at ingest time.
|
||||
2. **Derived semantics live elsewhere.** Policy Engine overlays, Vuln Explorer composition, and downstream reporting layers attach severity, precedence, policy verdicts, and UI hints.
|
||||
3. **Provenance is mandatory.** Every ingestion write must include original source metadata, digests, and signing/provenance evidence when available. Reject writes lacking provenance.
|
||||
4. **Deterministic outputs.** Given the same inputs, ingestion must produce identical documents, hashes, and event payloads across reruns.
|
||||
5. **Guardrails everywhere.** Roslyn analyzers, schema validators, and CI smoke tests should fail builds that attempt forbidden writes.
|
||||
|
||||
For detailed roles and ownership boundaries, see `AGENTS.md` at the repo root and the module-specific `ARCHITECTURE_*.md` dossiers.
|
||||
|
||||
Need the full contract? Read the [Aggregation-Only Contract reference](../ingestion/aggregation-only-contract.md) for schemas, error codes, and migration guidance.
|
||||
|
||||
@@ -41,9 +41,9 @@ Net result: partners and internal teams integrate quickly without reverse‑engi
|
||||
|
||||
### 3.1 Source of truth and layout
|
||||
|
||||
* Each service owns a **module-scoped OAS** file: `src/StellaOps.Api.OpenApi/<service>/openapi.yaml`.
|
||||
* Authority authentication/token surface now lives at `src/StellaOps.Api.OpenApi/authority/openapi.yaml`, covering `/token`, `/introspect`, `/revoke`, and `/jwks` flows with examples and scope catalog metadata.
|
||||
* An aggregate spec `src/StellaOps.Api.OpenApi/stella.yaml` is produced by build tooling that composes per-service specs, resolves `$ref`s, and validates cross-service schemas.
|
||||
* Each service owns a **module-scoped OAS** file: `src/Api/StellaOps.Api.OpenApi/<service>/openapi.yaml`.
|
||||
* Authority authentication/token surface now lives at `src/Api/StellaOps.Api.OpenApi/authority/openapi.yaml`, covering `/token`, `/introspect`, `/revoke`, and `/jwks` flows with examples and scope catalog metadata.
|
||||
* An aggregate spec `src/Api/StellaOps.Api.OpenApi/stella.yaml` is produced by build tooling that composes per-service specs, resolves `$ref`s, and validates cross-service schemas.
|
||||
* JSON Schema dialect: 2020‑12 (OpenAPI 3.1). No vendor‑specific features for core models.
|
||||
* Every response and error has at least one **validated example**.
|
||||
|
||||
@@ -138,13 +138,13 @@ Net result: partners and internal teams integrate quickly without reverse‑engi
|
||||
|
||||
### 4.1 New modules
|
||||
|
||||
* `src/StellaOps.Api.OpenApi/*` per service and aggregate composer
|
||||
* `src/StellaOps.Api.Governance` OAS linter rules and compatibility checker
|
||||
* `src/StellaOps.Sdk.Generator` codegen drivers, post‑processing templates, smoke tests
|
||||
* `src/StellaOps.Sdk.Release` packaging, signing, publishing
|
||||
* `src/StellaOps.DevPortal.Site` static generator and assets
|
||||
* `src/Api/StellaOps.Api.OpenApi/*` per service and aggregate composer
|
||||
* `src/Api/StellaOps.Api.Governance` OAS linter rules and compatibility checker
|
||||
* `src/Sdk/StellaOps.Sdk.Generator` codegen drivers, post‑processing templates, smoke tests
|
||||
* `src/Sdk/StellaOps.Sdk.Release` packaging, signing, publishing
|
||||
* `src/DevPortal/StellaOps.DevPortal.Site` static generator and assets
|
||||
* `test/contract` mock server config, golden examples
|
||||
* `src/StellaOps.ExportCenter.DevPortalOffline` bundler
|
||||
* `src/ExportCenter/StellaOps.ExportCenter.DevPortalOffline` bundler
|
||||
|
||||
### 4.2 Build flow
|
||||
|
||||
@@ -254,7 +254,7 @@ Add the banner at the top of each page:
|
||||
|
||||
**OAS & governance**
|
||||
|
||||
* [ ] Create `src/StellaOps.Api.OpenApi/<service>/openapi.yaml` for all services with minimal paths and shared components.
|
||||
* [ ] Create `src/Api/StellaOps.Api.OpenApi/<service>/openapi.yaml` for all services with minimal paths and shared components.
|
||||
* [ ] Implement aggregate composer and `$ref` resolver.
|
||||
* [ ] Add CI job: lint, validate, compatibility diff; block merges on failure.
|
||||
* [ ] Migrate all endpoints to standard error envelope and provide examples.
|
||||
|
||||
@@ -230,7 +230,7 @@ Slim wrapper used by CLI; returns 204 on success or `ERR_POL_001` payload.
|
||||
|
||||
## 6 · Run & Simulation APIs
|
||||
|
||||
> Schema reference: canonical policy run request/status/diff payloads ship with the Scheduler Models guide (`src/StellaOps.Scheduler.Models/docs/SCHED-MODELS-20-001-POLICY-RUNS.md`) and JSON fixtures under `samples/api/scheduler/policy-*.json`.
|
||||
> Schema reference: canonical policy run request/status/diff payloads ship with the Scheduler Models guide (`src/Scheduler/__Libraries/StellaOps.Scheduler.Models/docs/SCHED-MODELS-20-001-POLICY-RUNS.md`) and JSON fixtures under `samples/api/scheduler/policy-*.json`.
|
||||
|
||||
### 6.1 Trigger Run
|
||||
|
||||
@@ -389,7 +389,7 @@ Returns rule hit sequence:
|
||||
|
||||
## 9 · Compliance Checklist
|
||||
|
||||
- [ ] **Scopes enforced:** Endpoint access requires correct Authority scope mapping (see `/src/StellaOps.Authority/TASKS.md`).
|
||||
- [ ] **Scopes enforced:** Endpoint access requires correct Authority scope mapping (see `/src/Authority/StellaOps.Authority/TASKS.md`).
|
||||
- [ ] **Schemas current:** JSON examples align with Scheduler Models (`SCHED-MODELS-20-001`) and Policy Engine DTOs; update when contracts change.
|
||||
- [ ] **Error codes mapped:** `ERR_POL_*` table reflects implementation and CI tests cover edge cases.
|
||||
- [ ] **Pagination documented:** List endpoints specify page/size and cursor semantics; responses include `X-Total-Count` or `nextCursor`.
|
||||
|
||||
@@ -22,7 +22,7 @@ Non-goals: authoring ingestion logic, mutating Policy overlays, exposing interna
|
||||
|
||||
## 2 · Workspace & Packages
|
||||
|
||||
The console is implemented in `src/StellaOps.Web`, an Angular 17 workspace built on standalone components and Signals.
|
||||
The console is implemented in `src/Web/StellaOps.Web`, an Angular 17 workspace built on standalone components and Signals.
|
||||
|
||||
| Path | Purpose | Highlights |
|
||||
|------|---------|------------|
|
||||
@@ -148,7 +148,7 @@ Optimisation levers:
|
||||
## 6 · Offline & Configuration Workflows
|
||||
|
||||
- **Config manifest:** `/config.json` includes Authority issuer/client ID, gateway base URL, feature flags, telemetry endpoints, and offline hints. Operators can swap config by copying `src/config/config.sample.json` and editing before build, or by rewriting the response at gateway runtime.
|
||||
- **Deterministic install:** Documented in `src/StellaOps.Web/docs/DeterministicInstall.md`—`npm run ci:install` plus Chromium provisioning ensures offline runners reproduce builds.
|
||||
- **Deterministic install:** Documented in `src/Web/StellaOps.Web/docs/DeterministicInstall.md`—`npm run ci:install` plus Chromium provisioning ensures offline runners reproduce builds.
|
||||
- **Offline Kit parity:** UI validates downloads manifest signatures (cosign) and surfaces snapshot timestamps per tenant. When offline, buttons switch to CLI snippets (`stella runs export`, `stella downloads sync`).
|
||||
- **Feature flags:** `CONSOLE_FEATURE_FLAGS` toggles modules (`runs`, `downloads`, `telemetry`); offline bundles include flag manifest so UI can render only supported panes.
|
||||
- **Snapshot awareness:** Global banner shows snapshot timestamp and disables actions needing Authority fresh-auth when running in sealed mode.
|
||||
|
||||
@@ -1,168 +1,168 @@
|
||||
# StellaOps Architecture Overview (Sprint 19)
|
||||
|
||||
> **Ownership:** Architecture Guild • Docs Guild
|
||||
> **Audience:** Service owners, platform engineers, solution architects
|
||||
> **Related:** [High-Level Architecture](../07_HIGH_LEVEL_ARCHITECTURE.md), [Concelier Architecture](../ARCHITECTURE_CONCELIER.md), [Policy Engine Architecture](policy-engine.md), [Aggregation-Only Contract](../ingestion/aggregation-only-contract.md)
|
||||
|
||||
This dossier summarises the end-to-end runtime topology after the Aggregation-Only Contract (AOC) rollout. It highlights where raw facts live, how ingest services enforce guardrails, and how downstream components consume those facts to derive policy decisions and user-facing experiences.
|
||||
|
||||
---
|
||||
|
||||
## 1 · System landscape
|
||||
|
||||
```mermaid
|
||||
graph TD
|
||||
subgraph Edge["Clients & Automation"]
|
||||
CLI[stella CLI]
|
||||
UI[Console SPA]
|
||||
APIClients[CI / API Clients]
|
||||
end
|
||||
Gateway[API Gateway<br/>(JWT + DPoP scopes)]
|
||||
subgraph Scanner["Fact Collection"]
|
||||
ScannerWeb[Scanner.WebService]
|
||||
ScannerWorkers[Scanner.Workers]
|
||||
Agent[Agent Runtime]
|
||||
end
|
||||
subgraph Ingestion["Aggregation-Only Ingestion (AOC)"]
|
||||
Concelier[Concelier.WebService]
|
||||
Excititor[Excititor.WebService]
|
||||
RawStore[(MongoDB<br/>advisory_raw / vex_raw)]
|
||||
end
|
||||
subgraph Derivation["Policy & Overlay"]
|
||||
Policy[Policy Engine]
|
||||
Scheduler[Scheduler Services]
|
||||
Notify[Notifier]
|
||||
end
|
||||
subgraph Experience["UX & Export"]
|
||||
UIService[Console Backend]
|
||||
Exporters[Export / Offline Kit]
|
||||
end
|
||||
Observability[Telemetry Stack]
|
||||
|
||||
CLI --> Gateway
|
||||
UI --> Gateway
|
||||
APIClients --> Gateway
|
||||
Gateway --> ScannerWeb
|
||||
ScannerWeb --> ScannerWorkers
|
||||
ScannerWorkers --> Concelier
|
||||
ScannerWorkers --> Excititor
|
||||
Concelier --> RawStore
|
||||
Excititor --> RawStore
|
||||
RawStore --> Policy
|
||||
Policy --> Scheduler
|
||||
Policy --> Notify
|
||||
Policy --> UIService
|
||||
Scheduler --> UIService
|
||||
UIService --> Exporters
|
||||
Exporters --> CLI
|
||||
Exporters --> Offline[Offline Kit]
|
||||
Observability -.-> ScannerWeb
|
||||
Observability -.-> Concelier
|
||||
Observability -.-> Excititor
|
||||
Observability -.-> Policy
|
||||
Observability -.-> Scheduler
|
||||
Observability -.-> Notify
|
||||
```
|
||||
|
||||
Key boundaries:
|
||||
|
||||
- **AOC border.** Everything inside the Ingestion subgraph writes only immutable raw facts plus link hints. Derived severity, consensus, and risk remain outside the border.
|
||||
- **Policy-only derivation.** Policy Engine materialises `effective_finding_*` collections and emits overlays; other services consume but never mutate them.
|
||||
- **Tenant enforcement.** Authority-issued DPoP scopes flow through Gateway to every service; raw stores and overlays include `tenant` strictly.
|
||||
|
||||
---
|
||||
|
||||
## 2 · Aggregation-Only Contract focus
|
||||
|
||||
### 2.1 Responsibilities at the boundary
|
||||
|
||||
| Area | Services | Responsibilities under AOC | Forbidden under AOC |
|
||||
|------|----------|-----------------------------|---------------------|
|
||||
| **Ingestion (Concelier / Excititor)** | `StellaOps.Concelier.WebService`, `StellaOps.Excititor.WebService` | Fetch upstream advisories/VEX, verify signatures, compute linksets, append immutable documents to `advisory_raw` / `vex_raw`, emit observability signals, expose raw read APIs. | Computing severity, consensus, suppressions, or policy hints; merging upstream sources into a single derived record; mutating existing documents. |
|
||||
| **Policy & Overlay** | `StellaOps.Policy.Engine`, Scheduler | Join SBOM inventory with raw advisories/VEX, evaluate policies, issue `effective_finding_*` overlays, drive remediation workflows. | Writing to raw collections; bypassing guard scopes; running without recorded provenance. |
|
||||
| **Experience layers** | Console, CLI, Exporters | Surface raw facts + policy overlays; run `stella aoc verify`; render AOC dashboards and reports. | Accepting ingestion payloads that lack provenance or violate guard results. |
|
||||
|
||||
### 2.2 Raw stores
|
||||
|
||||
| Collection | Purpose | Key fields | Notes |
|
||||
|------------|---------|------------|-------|
|
||||
| `advisory_raw` | Immutable vendor/ecosystem advisory documents. | `_id`, `tenant`, `source.*`, `upstream.*`, `content.raw`, `linkset`, `supersedes`. | Idempotent by `(source.vendor, upstream.upstream_id, upstream.content_hash)`. |
|
||||
| `vex_raw` | Immutable vendor VEX statements. | Mirrors `advisory_raw`; `identifiers.statements` summarises affected components. | Maintains supersedes chain identical to advisory flow. |
|
||||
| Change streams (`advisory_raw_stream`, `vex_raw_stream`) | Feed Policy Engine and Scheduler. | `operationType`, `documentKey`, `fullDocument`, `tenant`, `traceId`. | Scope filtered per tenant before delivery. |
|
||||
|
||||
### 2.3 Guarded ingestion sequence
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant Upstream as Upstream Source
|
||||
participant Connector as Concelier/Excititor Connector
|
||||
participant Guard as AOCWriteGuard
|
||||
participant Mongo as MongoDB (advisory_raw / vex_raw)
|
||||
participant Stream as Change Stream
|
||||
participant Policy as Policy Engine
|
||||
|
||||
Upstream-->>Connector: CSAF / OSV / VEX document
|
||||
Connector->>Connector: Normalize transport, compute content_hash
|
||||
Connector->>Guard: Candidate raw doc (source + upstream + content + linkset)
|
||||
Guard-->>Connector: ERR_AOC_00x on violation
|
||||
Guard->>Mongo: Append immutable document (with tenant & supersedes)
|
||||
Mongo-->>Stream: Change event (tenant scoped)
|
||||
Stream->>Policy: Raw delta payload
|
||||
Policy->>Policy: Evaluate policies, compute effective findings
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 2.4 Authority scopes & tenancy
|
||||
|
||||
| Scope | Holder | Purpose | Notes |
|
||||
|-------|--------|---------|-------|
|
||||
| `advisory:ingest` / `vex:ingest` | Concelier / Excititor collectors | Append raw documents through ingestion endpoints. | Paired with tenant claims; requests without tenant are rejected. |
|
||||
| `advisory:read` / `vex:read` | DevOps verify identity, CLI | Run `stella aoc verify` or call `/aoc/verify`. | Read-only; cannot mutate raw docs. |
|
||||
| `effective:write` | Policy Engine | Materialise `effective_finding_*` overlays. | Only Policy Engine identity may hold; ingestion contexts receive `ERR_AOC_006` if they attempt. |
|
||||
| `findings:read` | Console, CLI, exports | Consume derived findings. | Enforced by Gateway and downstream services. |
|
||||
|
||||
---
|
||||
|
||||
## 3 · Data & control flow highlights
|
||||
|
||||
1. **Ingestion:** Concelier / Excititor connectors fetch upstream documents, compute linksets, and hand payloads to `AOCWriteGuard`. Guards validate schema, provenance, forbidden fields, supersedes pointers, and append-only rules before writing to Mongo.
|
||||
2. **Verification:** `stella aoc verify` (CLI/CI) and `/aoc/verify` endpoints replay guard checks against stored documents, mapping `ERR_AOC_00x` codes to exit codes for automation.
|
||||
3. **Policy evaluation:** Mongo change streams deliver tenant-scoped raw deltas. Policy Engine joins SBOM inventory (via BOM Index), executes deterministic policies, writes overlays, and emits events to Scheduler/Notify.
|
||||
4. **Experience surfaces:** Console renders an AOC dashboard showing ingestion latency, guard violations, and supersedes depth. CLI exposes raw-document fetch helpers for auditing. Offline Kit bundles raw collections alongside guard configs to keep air-gapped installs verifiable.
|
||||
5. **Observability:** All services emit `ingestion_write_total`, `aoc_violation_total{code}`, `ingestion_latency_seconds`, and trace spans `ingest.fetch`, `ingest.transform`, `ingest.write`, `aoc.guard`. Logs correlate via `traceId`, `tenant`, `source.vendor`, and `content_hash`.
|
||||
|
||||
---
|
||||
|
||||
## 4 · Offline & disaster readiness
|
||||
|
||||
- **Offline Kit:** Packages raw Mongo snapshots (`advisory_raw`, `vex_raw`) plus guard configuration and CLI verifier binaries so air-gapped sites can re-run AOC checks before promotion.
|
||||
- **Recovery:** Supersedes chains allow rollback to prior revisions without mutating documents. Disaster exercises must rehearse restoring from snapshot, replaying change streams into Policy Engine, and re-validating guard compliance.
|
||||
- **Migration:** Legacy normalised fields are moved to temporary views during cutover; ingestion runtime removes writes once guard-enforced path is live (see [Migration playbook](../ingestion/aggregation-only-contract.md#8-migration-playbook)).
|
||||
|
||||
---
|
||||
|
||||
## 5 · References
|
||||
|
||||
- [Aggregation-Only Contract reference](../ingestion/aggregation-only-contract.md)
|
||||
- [Concelier architecture](../ARCHITECTURE_CONCELIER.md)
|
||||
- [Excititor architecture](../ARCHITECTURE_EXCITITOR.md)
|
||||
- [Policy Engine architecture](policy-engine.md)
|
||||
- [Authority service](../ARCHITECTURE_AUTHORITY.md)
|
||||
- [Observability standards (upcoming)](../observability/policy.md) – interim reference for telemetry naming.
|
||||
|
||||
---
|
||||
|
||||
## 6 · Compliance checklist
|
||||
|
||||
- [ ] AOC guard enabled for all Concelier and Excititor write paths in production.
|
||||
- [ ] Mongo schema validators deployed for `advisory_raw` and `vex_raw`; change streams scoped per tenant.
|
||||
- [ ] Authority scopes (`advisory:*`, `vex:*`, `effective:*`) configured in Gateway and validated via integration tests.
|
||||
- [ ] `stella aoc verify` wired into CI/CD pipelines with seeded violation fixtures.
|
||||
- [ ] Console AOC dashboard and CLI documentation reference the new ingestion contract.
|
||||
- [ ] Offline Kit bundles include guard configs, verifier tooling, and documentation updates.
|
||||
- [ ] Observability dashboards include violation, latency, and supersedes depth metrics with alert thresholds.
|
||||
|
||||
---
|
||||
|
||||
*Last updated: 2025-10-26 (Sprint 19).*
|
||||
# StellaOps Architecture Overview (Sprint 19)
|
||||
|
||||
> **Ownership:** Architecture Guild • Docs Guild
|
||||
> **Audience:** Service owners, platform engineers, solution architects
|
||||
> **Related:** [High-Level Architecture](../07_HIGH_LEVEL_ARCHITECTURE.md), [Concelier Architecture](../ARCHITECTURE_CONCELIER.md), [Policy Engine Architecture](policy-engine.md), [Aggregation-Only Contract](../ingestion/aggregation-only-contract.md)
|
||||
|
||||
This dossier summarises the end-to-end runtime topology after the Aggregation-Only Contract (AOC) rollout. It highlights where raw facts live, how ingest services enforce guardrails, and how downstream components consume those facts to derive policy decisions and user-facing experiences.
|
||||
|
||||
---
|
||||
|
||||
## 1 · System landscape
|
||||
|
||||
```mermaid
|
||||
graph TD
|
||||
subgraph Edge["Clients & Automation"]
|
||||
CLI[stella CLI]
|
||||
UI[Console SPA]
|
||||
APIClients[CI / API Clients]
|
||||
end
|
||||
Gateway[API Gateway<br/>(JWT + DPoP scopes)]
|
||||
subgraph Scanner["Fact Collection"]
|
||||
ScannerWeb[Scanner.WebService]
|
||||
ScannerWorkers[Scanner.Workers]
|
||||
Agent[Agent Runtime]
|
||||
end
|
||||
subgraph Ingestion["Aggregation-Only Ingestion (AOC)"]
|
||||
Concelier[Concelier.WebService]
|
||||
Excititor[Excititor.WebService]
|
||||
RawStore[(MongoDB<br/>advisory_raw / vex_raw)]
|
||||
end
|
||||
subgraph Derivation["Policy & Overlay"]
|
||||
Policy[Policy Engine]
|
||||
Scheduler[Scheduler Services]
|
||||
Notify[Notifier]
|
||||
end
|
||||
subgraph Experience["UX & Export"]
|
||||
UIService[Console Backend]
|
||||
Exporters[Export / Offline Kit]
|
||||
end
|
||||
Observability[Telemetry Stack]
|
||||
|
||||
CLI --> Gateway
|
||||
UI --> Gateway
|
||||
APIClients --> Gateway
|
||||
Gateway --> ScannerWeb
|
||||
ScannerWeb --> ScannerWorkers
|
||||
ScannerWorkers --> Concelier
|
||||
ScannerWorkers --> Excititor
|
||||
Concelier --> RawStore
|
||||
Excititor --> RawStore
|
||||
RawStore --> Policy
|
||||
Policy --> Scheduler
|
||||
Policy --> Notify
|
||||
Policy --> UIService
|
||||
Scheduler --> UIService
|
||||
UIService --> Exporters
|
||||
Exporters --> CLI
|
||||
Exporters --> Offline[Offline Kit]
|
||||
Observability -.-> ScannerWeb
|
||||
Observability -.-> Concelier
|
||||
Observability -.-> Excititor
|
||||
Observability -.-> Policy
|
||||
Observability -.-> Scheduler
|
||||
Observability -.-> Notify
|
||||
```
|
||||
|
||||
Key boundaries:
|
||||
|
||||
- **AOC border.** Everything inside the Ingestion subgraph writes only immutable raw facts plus link hints. Derived severity, consensus, and risk remain outside the border.
|
||||
- **Policy-only derivation.** Policy Engine materialises `effective_finding_*` collections and emits overlays; other services consume but never mutate them.
|
||||
- **Tenant enforcement.** Authority-issued DPoP scopes flow through Gateway to every service; raw stores and overlays include `tenant` strictly.
|
||||
|
||||
---
|
||||
|
||||
## 2 · Aggregation-Only Contract focus
|
||||
|
||||
### 2.1 Responsibilities at the boundary
|
||||
|
||||
| Area | Services | Responsibilities under AOC | Forbidden under AOC |
|
||||
|------|----------|-----------------------------|---------------------|
|
||||
| **Ingestion (Concelier / Excititor)** | `StellaOps.Concelier.WebService`, `StellaOps.Excititor.WebService` | Fetch upstream advisories/VEX, verify signatures, compute linksets, append immutable documents to `advisory_raw` / `vex_raw`, emit observability signals, expose raw read APIs. | Computing severity, consensus, suppressions, or policy hints; merging upstream sources into a single derived record; mutating existing documents. |
|
||||
| **Policy & Overlay** | `StellaOps.Policy.Engine`, Scheduler | Join SBOM inventory with raw advisories/VEX, evaluate policies, issue `effective_finding_*` overlays, drive remediation workflows. | Writing to raw collections; bypassing guard scopes; running without recorded provenance. |
|
||||
| **Experience layers** | Console, CLI, Exporters | Surface raw facts + policy overlays; run `stella aoc verify`; render AOC dashboards and reports. | Accepting ingestion payloads that lack provenance or violate guard results. |
|
||||
|
||||
### 2.2 Raw stores
|
||||
|
||||
| Collection | Purpose | Key fields | Notes |
|
||||
|------------|---------|------------|-------|
|
||||
| `advisory_raw` | Immutable vendor/ecosystem advisory documents. | `_id`, `tenant`, `source.*`, `upstream.*`, `content.raw`, `linkset`, `supersedes`. | Idempotent by `(source.vendor, upstream.upstream_id, upstream.content_hash)`. |
|
||||
| `vex_raw` | Immutable vendor VEX statements. | Mirrors `advisory_raw`; `identifiers.statements` summarises affected components. | Maintains supersedes chain identical to advisory flow. |
|
||||
| Change streams (`advisory_raw_stream`, `vex_raw_stream`) | Feed Policy Engine and Scheduler. | `operationType`, `documentKey`, `fullDocument`, `tenant`, `traceId`. | Scope filtered per tenant before delivery. |
|
||||
|
||||
### 2.3 Guarded ingestion sequence
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant Upstream as Upstream Source
|
||||
participant Connector as Concelier/Excititor Connector
|
||||
participant Guard as AOCWriteGuard
|
||||
participant Mongo as MongoDB (advisory_raw / vex_raw)
|
||||
participant Stream as Change Stream
|
||||
participant Policy as Policy Engine
|
||||
|
||||
Upstream-->>Connector: CSAF / OSV / VEX document
|
||||
Connector->>Connector: Normalize transport, compute content_hash
|
||||
Connector->>Guard: Candidate raw doc (source + upstream + content + linkset)
|
||||
Guard-->>Connector: ERR_AOC_00x on violation
|
||||
Guard->>Mongo: Append immutable document (with tenant & supersedes)
|
||||
Mongo-->>Stream: Change event (tenant scoped)
|
||||
Stream->>Policy: Raw delta payload
|
||||
Policy->>Policy: Evaluate policies, compute effective findings
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 2.4 Authority scopes & tenancy
|
||||
|
||||
| Scope | Holder | Purpose | Notes |
|
||||
|-------|--------|---------|-------|
|
||||
| `advisory:ingest` / `vex:ingest` | Concelier / Excititor collectors | Append raw documents through ingestion endpoints. | Paired with tenant claims; requests without tenant are rejected. |
|
||||
| `advisory:read` / `vex:read` | DevOps verify identity, CLI | Run `stella aoc verify` or call `/aoc/verify`. | Read-only; cannot mutate raw docs. |
|
||||
| `effective:write` | Policy Engine | Materialise `effective_finding_*` overlays. | Only Policy Engine identity may hold; ingestion contexts receive `ERR_AOC_006` if they attempt. |
|
||||
| `findings:read` | Console, CLI, exports | Consume derived findings. | Enforced by Gateway and downstream services. |
|
||||
|
||||
---
|
||||
|
||||
## 3 · Data & control flow highlights
|
||||
|
||||
1. **Ingestion:** Concelier / Excititor connectors fetch upstream documents, compute linksets, and hand payloads to `AOCWriteGuard`. Guards validate schema, provenance, forbidden fields, supersedes pointers, and append-only rules before writing to Mongo.
|
||||
2. **Verification:** `stella aoc verify` (CLI/CI) and `/aoc/verify` endpoints replay guard checks against stored documents, mapping `ERR_AOC_00x` codes to exit codes for automation.
|
||||
3. **Policy evaluation:** Mongo change streams deliver tenant-scoped raw deltas. Policy Engine joins SBOM inventory (via BOM Index), executes deterministic policies, writes overlays, and emits events to Scheduler/Notify.
|
||||
4. **Experience surfaces:** Console renders an AOC dashboard showing ingestion latency, guard violations, and supersedes depth. CLI exposes raw-document fetch helpers for auditing. Offline Kit bundles raw collections alongside guard configs to keep air-gapped installs verifiable.
|
||||
5. **Observability:** All services emit `ingestion_write_total`, `aoc_violation_total{code}`, `ingestion_latency_seconds`, and trace spans `ingest.fetch`, `ingest.transform`, `ingest.write`, `aoc.guard`. Logs correlate via `traceId`, `tenant`, `source.vendor`, and `content_hash`.
|
||||
|
||||
---
|
||||
|
||||
## 4 · Offline & disaster readiness
|
||||
|
||||
- **Offline Kit:** Packages raw Mongo snapshots (`advisory_raw`, `vex_raw`) plus guard configuration and CLI verifier binaries so air-gapped sites can re-run AOC checks before promotion.
|
||||
- **Recovery:** Supersedes chains allow rollback to prior revisions without mutating documents. Disaster exercises must rehearse restoring from snapshot, replaying change streams into Policy Engine, and re-validating guard compliance.
|
||||
- **Migration:** Legacy normalised fields are moved to temporary views during cutover; ingestion runtime removes writes once guard-enforced path is live (see [Migration playbook](../ingestion/aggregation-only-contract.md#8-migration-playbook)).
|
||||
|
||||
---
|
||||
|
||||
## 5 · References
|
||||
|
||||
- [Aggregation-Only Contract reference](../ingestion/aggregation-only-contract.md)
|
||||
- [Concelier architecture](../ARCHITECTURE_CONCELIER.md)
|
||||
- [Excititor architecture](../ARCHITECTURE_EXCITITOR.md)
|
||||
- [Policy Engine architecture](policy-engine.md)
|
||||
- [Authority service](../ARCHITECTURE_AUTHORITY.md)
|
||||
- [Observability standards (upcoming)](../observability/policy.md) – interim reference for telemetry naming.
|
||||
|
||||
---
|
||||
|
||||
## 6 · Compliance checklist
|
||||
|
||||
- [ ] AOC guard enabled for all Concelier and Excititor write paths in production.
|
||||
- [ ] Mongo schema validators deployed for `advisory_raw` and `vex_raw`; change streams scoped per tenant.
|
||||
- [ ] Authority scopes (`advisory:*`, `vex:*`, `effective:*`) configured in Gateway and validated via integration tests.
|
||||
- [ ] `stella aoc verify` wired into CI/CD pipelines with seeded violation fixtures.
|
||||
- [ ] Console AOC dashboard and CLI documentation reference the new ingestion contract.
|
||||
- [ ] Offline Kit bundles include guard configs, verifier tooling, and documentation updates.
|
||||
- [ ] Observability dashboards include violation, latency, and supersedes depth metrics with alert thresholds.
|
||||
|
||||
---
|
||||
|
||||
*Last updated: 2025-10-26 (Sprint 19).*
|
||||
|
||||
@@ -1,243 +1,243 @@
|
||||
# Policy Engine Architecture (v2)
|
||||
|
||||
> **Ownership:** Policy Guild • Platform Guild
|
||||
> **Services:** `StellaOps.Policy.Engine` (Minimal API + worker host)
|
||||
> **Data Stores:** MongoDB (`policies`, `policy_runs`, `effective_finding_*`), Object storage (explain bundles), optional NATS/Mongo queue
|
||||
> **Related docs:** [Policy overview](../policy/overview.md), [DSL](../policy/dsl.md), [Lifecycle](../policy/lifecycle.md), [Runs](../policy/runs.md), [REST API](../api/policy.md), [Policy CLI](../cli/policy.md), [Architecture overview](../architecture/overview.md), [AOC reference](../ingestion/aggregation-only-contract.md)
|
||||
|
||||
This dossier describes the internal structure of the Policy Engine service delivered in Epic 2. It focuses on module boundaries, deterministic evaluation, orchestration, and integration contracts with Concelier, Excititor, SBOM Service, Authority, Scheduler, and Observability stacks.
|
||||
|
||||
The service operates strictly downstream of the **Aggregation-Only Contract (AOC)**. It consumes immutable `advisory_raw` and `vex_raw` documents emitted by Concelier and Excititor, derives findings inside Policy-owned collections, and never mutates ingestion stores. Refer to the architecture overview and AOC reference for system-wide guardrails and provenance obligations.
|
||||
|
||||
---
|
||||
|
||||
## 1 · Responsibilities & Constraints
|
||||
|
||||
- Compile and evaluate `stella-dsl@1` policy packs into deterministic verdicts.
|
||||
- Join SBOM inventory, Concelier advisories, and Excititor VEX evidence via canonical linksets and equivalence tables.
|
||||
- Materialise effective findings (`effective_finding_{policyId}`) with append-only history and produce explain traces.
|
||||
- Operate incrementally: react to change streams (advisory/vex/SBOM deltas) with ≤ 5 min SLA.
|
||||
- Provide simulations with diff summaries for UI/CLI workflows without modifying state.
|
||||
- Enforce strict determinism guard (no wall-clock, RNG, network beyond allow-listed services) and RBAC + tenancy via Authority scopes.
|
||||
- Support sealed/air-gapped deployments with offline bundles and sealed-mode hints.
|
||||
|
||||
Non-goals: policy authoring UI (handled by Console), ingestion or advisory normalisation (Concelier), VEX consensus (Excititor), runtime enforcement (Zastava).
|
||||
|
||||
---
|
||||
|
||||
## 2 · High-Level Architecture
|
||||
|
||||
```mermaid
|
||||
graph TD
|
||||
subgraph Clients
|
||||
CLI[stella CLI]
|
||||
UI[Console Policy Editor]
|
||||
CI[CI Pipelines]
|
||||
end
|
||||
subgraph PolicyEngine["StellaOps.Policy.Engine"]
|
||||
API[Minimal API Host]
|
||||
Orchestrator[Run Orchestrator]
|
||||
WorkerPool[Evaluation Workers]
|
||||
Compiler[DSL Compiler Cache]
|
||||
Materializer[Effective Findings Writer]
|
||||
end
|
||||
subgraph RawStores["Raw Stores (AOC)"]
|
||||
AdvisoryRaw[(MongoDB<br/>advisory_raw)]
|
||||
VexRaw[(MongoDB<br/>vex_raw)]
|
||||
end
|
||||
subgraph Derived["Derived Stores"]
|
||||
Mongo[(MongoDB<br/>policies / policy_runs / effective_finding_*)]
|
||||
Blob[(Object Store / Evidence Locker)]
|
||||
Queue[(Mongo Queue / NATS)]
|
||||
end
|
||||
Concelier[(Concelier APIs)]
|
||||
Excititor[(Excititor APIs)]
|
||||
SBOM[(SBOM Service)]
|
||||
Authority[(Authority / DPoP Gateway)]
|
||||
|
||||
CLI --> API
|
||||
UI --> API
|
||||
CI --> API
|
||||
API --> Compiler
|
||||
API --> Orchestrator
|
||||
Orchestrator --> Queue
|
||||
Queue --> WorkerPool
|
||||
Concelier --> AdvisoryRaw
|
||||
Excititor --> VexRaw
|
||||
WorkerPool --> AdvisoryRaw
|
||||
WorkerPool --> VexRaw
|
||||
WorkerPool --> SBOM
|
||||
WorkerPool --> Materializer
|
||||
Materializer --> Mongo
|
||||
WorkerPool --> Blob
|
||||
API --> Mongo
|
||||
API --> Blob
|
||||
API --> Authority
|
||||
Orchestrator --> Mongo
|
||||
Authority --> API
|
||||
```
|
||||
|
||||
Key notes:
|
||||
|
||||
- API host exposes lifecycle, run, simulate, findings endpoints with DPoP-bound OAuth enforcement.
|
||||
- Orchestrator manages run scheduling/fairness; writes run tickets to queue, leases jobs to worker pool.
|
||||
- Workers evaluate policies using cached IR; join external services via tenant-scoped clients; pull immutable advisories/VEX from the raw stores; write derived overlays to Mongo and optional explain bundles to blob storage.
|
||||
- Observability (metrics/traces/logs) integrated via OpenTelemetry (not shown).
|
||||
|
||||
---
|
||||
|
||||
### 2.1 · AOC inputs & immutability
|
||||
|
||||
- **Raw-only reads.** Evaluation workers access `advisory_raw` / `vex_raw` via tenant-scoped Mongo clients or the Concelier/Excititor raw APIs. No Policy Engine component is permitted to mutate these collections.
|
||||
- **Guarded ingestion.** `AOCWriteGuard` rejects forbidden fields before data reaches the raw stores. Policy tests replay known `ERR_AOC_00x` violations to confirm ingestion compliance.
|
||||
- **Change streams as contract.** Run orchestration stores resumable cursors for raw change streams. Replays of these cursors (e.g., after failover) must yield identical materialisation outcomes.
|
||||
- **Derived stores only.** All severity, consensus, and suppression state lives in `effective_finding_*` collections and explain bundles owned by Policy Engine. Provenance fields link back to raw document IDs so auditors can trace every verdict.
|
||||
- **Authority scopes.** Only the Policy Engine service identity holds `effective:write`. Ingestion identities retain `advisory:*`/`vex:*` scopes, ensuring separation of duties enforced by Authority and the API Gateway.
|
||||
|
||||
---
|
||||
|
||||
## 3 · Module Breakdown
|
||||
|
||||
| Module | Responsibility | Notes |
|
||||
|--------|----------------|-------|
|
||||
| **Configuration** (`Configuration/`) | Bind settings (Mongo URIs, queue options, service URLs, sealed mode), validate on start. | Strict schema; fails fast on missing secrets. |
|
||||
| **Authority Client** (`Authority/`) | Acquire tokens, enforce scopes, perform DPoP key rotation. | Only service identity uses `effective:write`. |
|
||||
| **DSL Compiler** (`Dsl/`) | Parse, canonicalise, IR generation, checksum caching. | Uses Roslyn-like pipeline; caches by `policyId+version+hash`. |
|
||||
| **Selection Layer** (`Selection/`) | Batch SBOM ↔ advisory ↔ VEX joiners; apply equivalence tables; support incremental cursors. | Deterministic ordering (SBOM → advisory → VEX). |
|
||||
| **Evaluator** (`Evaluation/`) | Execute IR with first-match semantics, compute severity/trust/reachability weights, record rule hits. | Stateless; all inputs provided by selection layer. |
|
||||
| **Materialiser** (`Materialization/`) | Upsert effective findings, append history, manage explain bundle exports. | Mongo transactions per SBOM chunk. |
|
||||
| **Orchestrator** (`Runs/`) | Change-stream ingestion, fairness, retry/backoff, queue writer. | Works with Scheduler Models DTOs. |
|
||||
| **API** (`Api/`) | Minimal API endpoints, DTO validation, problem responses, idempotency. | Generated clients for CLI/UI. |
|
||||
| **Observability** (`Telemetry/`) | Metrics (`policy_run_seconds`, `rules_fired_total`), traces, structured logs. | Sampled rule-hit logs with redaction. |
|
||||
| **Offline Adapter** (`Offline/`) | Bundle export/import (policies, simulations, runs), sealed-mode enforcement. | Uses DSSE signing via Signer service. |
|
||||
|
||||
---
|
||||
|
||||
## 4 · Data Model & Persistence
|
||||
|
||||
### 4.1 Collections
|
||||
|
||||
- `policies` – policy versions, metadata, lifecycle states, simulation artefact references.
|
||||
- `policy_runs` – run records, inputs (cursors, env), stats, determinism hash, run status.
|
||||
- `policy_run_events` – append-only log (queued, leased, completed, failed, canceled, replay).
|
||||
- `effective_finding_{policyId}` – current verdict snapshot per finding.
|
||||
- `effective_finding_{policyId}_history` – append-only history (previous verdicts, timestamps, runId).
|
||||
- `policy_reviews` – review comments/decisions.
|
||||
|
||||
### 4.2 Schema Highlights
|
||||
|
||||
- Run records include `changeDigests` (hash of advisory/VEX inputs) for replay verification.
|
||||
- Effective findings store provenance references (`advisory_raw_ids`, `vex_raw_ids`, `sbom_component_id`).
|
||||
- All collections include `tenant`, `policyId`, `version`, `createdAt`, `updatedAt`, `traceId` for audit.
|
||||
|
||||
### 4.3 Indexing
|
||||
|
||||
- Compound indexes: `{tenant, policyId, status}` on `policies`; `{tenant, policyId, status, startedAt}` on `policy_runs`; `{policyId, sbomId, findingKey}` on findings.
|
||||
- TTL indexes on transient explain bundle references (configurable).
|
||||
|
||||
---
|
||||
|
||||
## 5 · Evaluation Pipeline
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
autonumber
|
||||
participant Worker as EvaluationWorker
|
||||
participant Compiler as CompilerCache
|
||||
participant Selector as SelectionLayer
|
||||
participant Eval as Evaluator
|
||||
participant Mat as Materialiser
|
||||
participant Expl as ExplainStore
|
||||
|
||||
Worker->>Compiler: Load IR (policyId, version, digest)
|
||||
Compiler-->>Worker: CompiledPolicy (cached or compiled)
|
||||
Worker->>Selector: Fetch tuple batches (sbom, advisory, vex)
|
||||
Selector-->>Worker: Deterministic batches (1024 tuples)
|
||||
loop For each batch
|
||||
Worker->>Eval: Execute rules (batch, env)
|
||||
Eval-->>Worker: Verdicts + rule hits
|
||||
Worker->>Mat: Upsert effective findings
|
||||
Mat-->>Worker: Success
|
||||
Worker->>Expl: Persist sampled explain traces (optional)
|
||||
end
|
||||
Worker->>Mat: Append history + run stats
|
||||
Worker-->>Worker: Compute determinism hash
|
||||
Worker->>+Mat: Finalize transaction
|
||||
Mat-->>Worker: Ack
|
||||
```
|
||||
|
||||
Determinism guard instrumentation wraps the evaluator, rejecting access to forbidden APIs and ensuring batch ordering remains stable.
|
||||
|
||||
---
|
||||
|
||||
## 6 · Run Orchestration & Incremental Flow
|
||||
|
||||
- **Change streams:** Concelier and Excititor publish document changes to the scheduler queue (`policy.trigger.delta`). Payload includes `tenant`, `source`, `linkset digests`, `cursor`.
|
||||
- **Orchestrator:** Maintains per-tenant backlog; merges deltas until time/size thresholds met, then enqueues `PolicyRunRequest`.
|
||||
- **Queue:** Mongo queue with lease; each job assigned `leaseDuration`, `maxAttempts`.
|
||||
- **Workers:** Lease jobs, execute evaluation pipeline, report status (success/failure/canceled). Failures with recoverable errors requeue with backoff; determinism or schema violations mark job `failed` and raise incident event.
|
||||
- **Fairness:** Round-robin per `{tenant, policyId}`; emergency jobs (`priority=emergency`) jump queue but limited via circuit breaker.
|
||||
- **Replay:** On demand, orchestrator rehydrates run via stored cursors and exports sealed bundle for audit/CI determinism checks.
|
||||
|
||||
---
|
||||
|
||||
## 7 · Security & Tenancy
|
||||
|
||||
- **Auth:** All API calls pass through Authority gateway; DPoP tokens enforced for service-to-service (Policy Engine service principal). CLI/UI tokens include scope claims.
|
||||
- **Scopes:** Mutations require `policy:*` scopes corresponding to action; `effective:write` restricted to service identity.
|
||||
- **Tenancy:** All queries filter by `tenant`. Service identity uses `tenant-global` for shared policies; cross-tenant reads prohibited unless `policy:tenant-admin` scope present.
|
||||
- **Secrets:** Configuration loaded via environment variables or sealed secrets; runtime avoids writing secrets to logs.
|
||||
- **Determinism guard:** Static analyzer prevents referencing forbidden namespaces; runtime guard intercepts `DateTime.Now`, `Random`, `Guid`, HTTP clients beyond allow-list.
|
||||
- **Sealed mode:** Global flag disables outbound network except allow-listed internal hosts; watchers fail fast if unexpected egress attempted.
|
||||
|
||||
---
|
||||
|
||||
## 8 · Observability
|
||||
|
||||
- Metrics:
|
||||
- `policy_run_seconds{mode,tenant,policy}` (histogram)
|
||||
- `policy_run_queue_depth{tenant}`
|
||||
- `policy_rules_fired_total{policy,rule}`
|
||||
- `policy_vex_overrides_total{policy,vendor}`
|
||||
- Logs: Structured JSON with `traceId`, `policyId`, `version`, `runId`, `tenant`, `phase`. Guard ensures no sensitive data leakage.
|
||||
- Traces: Spans `policy.select`, `policy.evaluate`, `policy.materialize`, `policy.simulate`. Trace IDs surfaced to CLI/UI.
|
||||
- Incident mode toggles 100 % sampling and extended retention windows.
|
||||
|
||||
---
|
||||
|
||||
## 9 · Offline / Bundle Integration
|
||||
|
||||
- **Imports:** Offline Kit delivers policy packs, advisory/VEX snapshots, SBOM updates. Policy Engine ingests bundles via `offline import`.
|
||||
- **Exports:** `stella policy bundle export` packages policy, IR digest, simulations, run metadata; UI provides export triggers.
|
||||
- **Sealed hints:** Explain traces annotate when cached values used (EPSS, KEV). Run records mark `env.sealed=true`.
|
||||
- **Sync cadence:** Operators perform monthly bundle sync; Policy Engine warns when snapshots > configured staleness (default 14 days).
|
||||
|
||||
---
|
||||
|
||||
## 10 · Testing & Quality
|
||||
|
||||
- **Unit tests:** DSL parsing, evaluator semantics, guard enforcement.
|
||||
- **Integration tests:** Joiners with sample SBOM/advisory/VEX data; materialisation with deterministic ordering; API contract tests generated from OpenAPI.
|
||||
- **Property tests:** Ensure rule evaluation deterministic across permutations.
|
||||
- **Golden tests:** Replay recorded runs, compare determinism hash.
|
||||
- **Performance tests:** Evaluate 100k component / 1M advisory dataset under warmed caches (<30 s full run).
|
||||
- **Chaos hooks:** Optional toggles to simulate upstream latency/failures; used in staging.
|
||||
|
||||
---
|
||||
|
||||
## 11 · Compliance Checklist
|
||||
|
||||
- [ ] **Determinism guard enforced:** Static analyzer + runtime guard block wall-clock, RNG, unauthorized network calls.
|
||||
- [ ] **Incremental correctness:** Change-stream cursors stored and replayed during tests; unit/integration coverage for dedupe.
|
||||
- [ ] **RBAC validated:** Endpoint scope requirements match Authority configuration; integration tests cover deny/allow.
|
||||
- [ ] **AOC separation enforced:** No code path writes to `advisory_raw` / `vex_raw`; integration tests capture `ERR_AOC_00x` handling; read-only clients verified.
|
||||
- [ ] **Effective findings ownership:** Only Policy Engine identity holds `effective:write`; unauthorized callers receive `ERR_AOC_006`.
|
||||
- [ ] **Observability wired:** Metrics/traces/logs exported with correlation IDs; dashboards include `aoc_violation_total` and ingest latency panels.
|
||||
- [ ] **Offline parity:** Sealed-mode tests executed; bundle import/export flows documented and validated.
|
||||
- [ ] **Schema docs synced:** DTOs match Scheduler Models (`SCHED-MODELS-20-001`); JSON schemas committed.
|
||||
- [ ] **Security reviews complete:** Threat model (including queue poisoning, determinism bypass, data exfiltration) documented; mitigations in place.
|
||||
- [ ] **Disaster recovery rehearsed:** Run replay+rollback procedures tested and recorded.
|
||||
|
||||
---
|
||||
|
||||
*Last updated: 2025-10-26 (Sprint 19).*
|
||||
# Policy Engine Architecture (v2)
|
||||
|
||||
> **Ownership:** Policy Guild • Platform Guild
|
||||
> **Services:** `StellaOps.Policy.Engine` (Minimal API + worker host)
|
||||
> **Data Stores:** MongoDB (`policies`, `policy_runs`, `effective_finding_*`), Object storage (explain bundles), optional NATS/Mongo queue
|
||||
> **Related docs:** [Policy overview](../policy/overview.md), [DSL](../policy/dsl.md), [Lifecycle](../policy/lifecycle.md), [Runs](../policy/runs.md), [REST API](../api/policy.md), [Policy CLI](../cli/policy.md), [Architecture overview](../architecture/overview.md), [AOC reference](../ingestion/aggregation-only-contract.md)
|
||||
|
||||
This dossier describes the internal structure of the Policy Engine service delivered in Epic 2. It focuses on module boundaries, deterministic evaluation, orchestration, and integration contracts with Concelier, Excititor, SBOM Service, Authority, Scheduler, and Observability stacks.
|
||||
|
||||
The service operates strictly downstream of the **Aggregation-Only Contract (AOC)**. It consumes immutable `advisory_raw` and `vex_raw` documents emitted by Concelier and Excititor, derives findings inside Policy-owned collections, and never mutates ingestion stores. Refer to the architecture overview and AOC reference for system-wide guardrails and provenance obligations.
|
||||
|
||||
---
|
||||
|
||||
## 1 · Responsibilities & Constraints
|
||||
|
||||
- Compile and evaluate `stella-dsl@1` policy packs into deterministic verdicts.
|
||||
- Join SBOM inventory, Concelier advisories, and Excititor VEX evidence via canonical linksets and equivalence tables.
|
||||
- Materialise effective findings (`effective_finding_{policyId}`) with append-only history and produce explain traces.
|
||||
- Operate incrementally: react to change streams (advisory/vex/SBOM deltas) with ≤ 5 min SLA.
|
||||
- Provide simulations with diff summaries for UI/CLI workflows without modifying state.
|
||||
- Enforce strict determinism guard (no wall-clock, RNG, network beyond allow-listed services) and RBAC + tenancy via Authority scopes.
|
||||
- Support sealed/air-gapped deployments with offline bundles and sealed-mode hints.
|
||||
|
||||
Non-goals: policy authoring UI (handled by Console), ingestion or advisory normalisation (Concelier), VEX consensus (Excititor), runtime enforcement (Zastava).
|
||||
|
||||
---
|
||||
|
||||
## 2 · High-Level Architecture
|
||||
|
||||
```mermaid
|
||||
graph TD
|
||||
subgraph Clients
|
||||
CLI[stella CLI]
|
||||
UI[Console Policy Editor]
|
||||
CI[CI Pipelines]
|
||||
end
|
||||
subgraph PolicyEngine["StellaOps.Policy.Engine"]
|
||||
API[Minimal API Host]
|
||||
Orchestrator[Run Orchestrator]
|
||||
WorkerPool[Evaluation Workers]
|
||||
Compiler[DSL Compiler Cache]
|
||||
Materializer[Effective Findings Writer]
|
||||
end
|
||||
subgraph RawStores["Raw Stores (AOC)"]
|
||||
AdvisoryRaw[(MongoDB<br/>advisory_raw)]
|
||||
VexRaw[(MongoDB<br/>vex_raw)]
|
||||
end
|
||||
subgraph Derived["Derived Stores"]
|
||||
Mongo[(MongoDB<br/>policies / policy_runs / effective_finding_*)]
|
||||
Blob[(Object Store / Evidence Locker)]
|
||||
Queue[(Mongo Queue / NATS)]
|
||||
end
|
||||
Concelier[(Concelier APIs)]
|
||||
Excititor[(Excititor APIs)]
|
||||
SBOM[(SBOM Service)]
|
||||
Authority[(Authority / DPoP Gateway)]
|
||||
|
||||
CLI --> API
|
||||
UI --> API
|
||||
CI --> API
|
||||
API --> Compiler
|
||||
API --> Orchestrator
|
||||
Orchestrator --> Queue
|
||||
Queue --> WorkerPool
|
||||
Concelier --> AdvisoryRaw
|
||||
Excititor --> VexRaw
|
||||
WorkerPool --> AdvisoryRaw
|
||||
WorkerPool --> VexRaw
|
||||
WorkerPool --> SBOM
|
||||
WorkerPool --> Materializer
|
||||
Materializer --> Mongo
|
||||
WorkerPool --> Blob
|
||||
API --> Mongo
|
||||
API --> Blob
|
||||
API --> Authority
|
||||
Orchestrator --> Mongo
|
||||
Authority --> API
|
||||
```
|
||||
|
||||
Key notes:
|
||||
|
||||
- API host exposes lifecycle, run, simulate, findings endpoints with DPoP-bound OAuth enforcement.
|
||||
- Orchestrator manages run scheduling/fairness; writes run tickets to queue, leases jobs to worker pool.
|
||||
- Workers evaluate policies using cached IR; join external services via tenant-scoped clients; pull immutable advisories/VEX from the raw stores; write derived overlays to Mongo and optional explain bundles to blob storage.
|
||||
- Observability (metrics/traces/logs) integrated via OpenTelemetry (not shown).
|
||||
|
||||
---
|
||||
|
||||
### 2.1 · AOC inputs & immutability
|
||||
|
||||
- **Raw-only reads.** Evaluation workers access `advisory_raw` / `vex_raw` via tenant-scoped Mongo clients or the Concelier/Excititor raw APIs. No Policy Engine component is permitted to mutate these collections.
|
||||
- **Guarded ingestion.** `AOCWriteGuard` rejects forbidden fields before data reaches the raw stores. Policy tests replay known `ERR_AOC_00x` violations to confirm ingestion compliance.
|
||||
- **Change streams as contract.** Run orchestration stores resumable cursors for raw change streams. Replays of these cursors (e.g., after failover) must yield identical materialisation outcomes.
|
||||
- **Derived stores only.** All severity, consensus, and suppression state lives in `effective_finding_*` collections and explain bundles owned by Policy Engine. Provenance fields link back to raw document IDs so auditors can trace every verdict.
|
||||
- **Authority scopes.** Only the Policy Engine service identity holds `effective:write`. Ingestion identities retain `advisory:*`/`vex:*` scopes, ensuring separation of duties enforced by Authority and the API Gateway.
|
||||
|
||||
---
|
||||
|
||||
## 3 · Module Breakdown
|
||||
|
||||
| Module | Responsibility | Notes |
|
||||
|--------|----------------|-------|
|
||||
| **Configuration** (`Configuration/`) | Bind settings (Mongo URIs, queue options, service URLs, sealed mode), validate on start. | Strict schema; fails fast on missing secrets. |
|
||||
| **Authority Client** (`Authority/`) | Acquire tokens, enforce scopes, perform DPoP key rotation. | Only service identity uses `effective:write`. |
|
||||
| **DSL Compiler** (`Dsl/`) | Parse, canonicalise, IR generation, checksum caching. | Uses Roslyn-like pipeline; caches by `policyId+version+hash`. |
|
||||
| **Selection Layer** (`Selection/`) | Batch SBOM ↔ advisory ↔ VEX joiners; apply equivalence tables; support incremental cursors. | Deterministic ordering (SBOM → advisory → VEX). |
|
||||
| **Evaluator** (`Evaluation/`) | Execute IR with first-match semantics, compute severity/trust/reachability weights, record rule hits. | Stateless; all inputs provided by selection layer. |
|
||||
| **Materialiser** (`Materialization/`) | Upsert effective findings, append history, manage explain bundle exports. | Mongo transactions per SBOM chunk. |
|
||||
| **Orchestrator** (`Runs/`) | Change-stream ingestion, fairness, retry/backoff, queue writer. | Works with Scheduler Models DTOs. |
|
||||
| **API** (`Api/`) | Minimal API endpoints, DTO validation, problem responses, idempotency. | Generated clients for CLI/UI. |
|
||||
| **Observability** (`Telemetry/`) | Metrics (`policy_run_seconds`, `rules_fired_total`), traces, structured logs. | Sampled rule-hit logs with redaction. |
|
||||
| **Offline Adapter** (`Offline/`) | Bundle export/import (policies, simulations, runs), sealed-mode enforcement. | Uses DSSE signing via Signer service. |
|
||||
|
||||
---
|
||||
|
||||
## 4 · Data Model & Persistence
|
||||
|
||||
### 4.1 Collections
|
||||
|
||||
- `policies` – policy versions, metadata, lifecycle states, simulation artefact references.
|
||||
- `policy_runs` – run records, inputs (cursors, env), stats, determinism hash, run status.
|
||||
- `policy_run_events` – append-only log (queued, leased, completed, failed, canceled, replay).
|
||||
- `effective_finding_{policyId}` – current verdict snapshot per finding.
|
||||
- `effective_finding_{policyId}_history` – append-only history (previous verdicts, timestamps, runId).
|
||||
- `policy_reviews` – review comments/decisions.
|
||||
|
||||
### 4.2 Schema Highlights
|
||||
|
||||
- Run records include `changeDigests` (hash of advisory/VEX inputs) for replay verification.
|
||||
- Effective findings store provenance references (`advisory_raw_ids`, `vex_raw_ids`, `sbom_component_id`).
|
||||
- All collections include `tenant`, `policyId`, `version`, `createdAt`, `updatedAt`, `traceId` for audit.
|
||||
|
||||
### 4.3 Indexing
|
||||
|
||||
- Compound indexes: `{tenant, policyId, status}` on `policies`; `{tenant, policyId, status, startedAt}` on `policy_runs`; `{policyId, sbomId, findingKey}` on findings.
|
||||
- TTL indexes on transient explain bundle references (configurable).
|
||||
|
||||
---
|
||||
|
||||
## 5 · Evaluation Pipeline
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
autonumber
|
||||
participant Worker as EvaluationWorker
|
||||
participant Compiler as CompilerCache
|
||||
participant Selector as SelectionLayer
|
||||
participant Eval as Evaluator
|
||||
participant Mat as Materialiser
|
||||
participant Expl as ExplainStore
|
||||
|
||||
Worker->>Compiler: Load IR (policyId, version, digest)
|
||||
Compiler-->>Worker: CompiledPolicy (cached or compiled)
|
||||
Worker->>Selector: Fetch tuple batches (sbom, advisory, vex)
|
||||
Selector-->>Worker: Deterministic batches (1024 tuples)
|
||||
loop For each batch
|
||||
Worker->>Eval: Execute rules (batch, env)
|
||||
Eval-->>Worker: Verdicts + rule hits
|
||||
Worker->>Mat: Upsert effective findings
|
||||
Mat-->>Worker: Success
|
||||
Worker->>Expl: Persist sampled explain traces (optional)
|
||||
end
|
||||
Worker->>Mat: Append history + run stats
|
||||
Worker-->>Worker: Compute determinism hash
|
||||
Worker->>+Mat: Finalize transaction
|
||||
Mat-->>Worker: Ack
|
||||
```
|
||||
|
||||
Determinism guard instrumentation wraps the evaluator, rejecting access to forbidden APIs and ensuring batch ordering remains stable.
|
||||
|
||||
---
|
||||
|
||||
## 6 · Run Orchestration & Incremental Flow
|
||||
|
||||
- **Change streams:** Concelier and Excititor publish document changes to the scheduler queue (`policy.trigger.delta`). Payload includes `tenant`, `source`, `linkset digests`, `cursor`.
|
||||
- **Orchestrator:** Maintains per-tenant backlog; merges deltas until time/size thresholds met, then enqueues `PolicyRunRequest`.
|
||||
- **Queue:** Mongo queue with lease; each job assigned `leaseDuration`, `maxAttempts`.
|
||||
- **Workers:** Lease jobs, execute evaluation pipeline, report status (success/failure/canceled). Failures with recoverable errors requeue with backoff; determinism or schema violations mark job `failed` and raise incident event.
|
||||
- **Fairness:** Round-robin per `{tenant, policyId}`; emergency jobs (`priority=emergency`) jump queue but limited via circuit breaker.
|
||||
- **Replay:** On demand, orchestrator rehydrates run via stored cursors and exports sealed bundle for audit/CI determinism checks.
|
||||
|
||||
---
|
||||
|
||||
## 7 · Security & Tenancy
|
||||
|
||||
- **Auth:** All API calls pass through Authority gateway; DPoP tokens enforced for service-to-service (Policy Engine service principal). CLI/UI tokens include scope claims.
|
||||
- **Scopes:** Mutations require `policy:*` scopes corresponding to action; `effective:write` restricted to service identity.
|
||||
- **Tenancy:** All queries filter by `tenant`. Service identity uses `tenant-global` for shared policies; cross-tenant reads prohibited unless `policy:tenant-admin` scope present.
|
||||
- **Secrets:** Configuration loaded via environment variables or sealed secrets; runtime avoids writing secrets to logs.
|
||||
- **Determinism guard:** Static analyzer prevents referencing forbidden namespaces; runtime guard intercepts `DateTime.Now`, `Random`, `Guid`, HTTP clients beyond allow-list.
|
||||
- **Sealed mode:** Global flag disables outbound network except allow-listed internal hosts; watchers fail fast if unexpected egress attempted.
|
||||
|
||||
---
|
||||
|
||||
## 8 · Observability
|
||||
|
||||
- Metrics:
|
||||
- `policy_run_seconds{mode,tenant,policy}` (histogram)
|
||||
- `policy_run_queue_depth{tenant}`
|
||||
- `policy_rules_fired_total{policy,rule}`
|
||||
- `policy_vex_overrides_total{policy,vendor}`
|
||||
- Logs: Structured JSON with `traceId`, `policyId`, `version`, `runId`, `tenant`, `phase`. Guard ensures no sensitive data leakage.
|
||||
- Traces: Spans `policy.select`, `policy.evaluate`, `policy.materialize`, `policy.simulate`. Trace IDs surfaced to CLI/UI.
|
||||
- Incident mode toggles 100 % sampling and extended retention windows.
|
||||
|
||||
---
|
||||
|
||||
## 9 · Offline / Bundle Integration
|
||||
|
||||
- **Imports:** Offline Kit delivers policy packs, advisory/VEX snapshots, SBOM updates. Policy Engine ingests bundles via `offline import`.
|
||||
- **Exports:** `stella policy bundle export` packages policy, IR digest, simulations, run metadata; UI provides export triggers.
|
||||
- **Sealed hints:** Explain traces annotate when cached values used (EPSS, KEV). Run records mark `env.sealed=true`.
|
||||
- **Sync cadence:** Operators perform monthly bundle sync; Policy Engine warns when snapshots > configured staleness (default 14 days).
|
||||
|
||||
---
|
||||
|
||||
## 10 · Testing & Quality
|
||||
|
||||
- **Unit tests:** DSL parsing, evaluator semantics, guard enforcement.
|
||||
- **Integration tests:** Joiners with sample SBOM/advisory/VEX data; materialisation with deterministic ordering; API contract tests generated from OpenAPI.
|
||||
- **Property tests:** Ensure rule evaluation deterministic across permutations.
|
||||
- **Golden tests:** Replay recorded runs, compare determinism hash.
|
||||
- **Performance tests:** Evaluate 100k component / 1M advisory dataset under warmed caches (<30 s full run).
|
||||
- **Chaos hooks:** Optional toggles to simulate upstream latency/failures; used in staging.
|
||||
|
||||
---
|
||||
|
||||
## 11 · Compliance Checklist
|
||||
|
||||
- [ ] **Determinism guard enforced:** Static analyzer + runtime guard block wall-clock, RNG, unauthorized network calls.
|
||||
- [ ] **Incremental correctness:** Change-stream cursors stored and replayed during tests; unit/integration coverage for dedupe.
|
||||
- [ ] **RBAC validated:** Endpoint scope requirements match Authority configuration; integration tests cover deny/allow.
|
||||
- [ ] **AOC separation enforced:** No code path writes to `advisory_raw` / `vex_raw`; integration tests capture `ERR_AOC_00x` handling; read-only clients verified.
|
||||
- [ ] **Effective findings ownership:** Only Policy Engine identity holds `effective:write`; unauthorized callers receive `ERR_AOC_006`.
|
||||
- [ ] **Observability wired:** Metrics/traces/logs exported with correlation IDs; dashboards include `aoc_violation_total` and ingest latency panels.
|
||||
- [ ] **Offline parity:** Sealed-mode tests executed; bundle import/export flows documented and validated.
|
||||
- [ ] **Schema docs synced:** DTOs match Scheduler Models (`SCHED-MODELS-20-001`); JSON schemas committed.
|
||||
- [ ] **Security reviews complete:** Threat model (including queue poisoning, determinism bypass, data exfiltration) documented; mitigations in place.
|
||||
- [ ] **Disaster recovery rehearsed:** Run replay+rollback procedures tested and recorded.
|
||||
|
||||
---
|
||||
|
||||
*Last updated: 2025-10-26 (Sprint 19).*
|
||||
|
||||
@@ -1,13 +1,13 @@
|
||||
# UI Tours Media Assets
|
||||
|
||||
Store annotated screenshots and GIFs referenced by `/docs/examples/ui-tours.md` in this directory. Use the naming convention documented in the guide (e.g., `triage-step-01.png`, `triage-flow.gif`).
|
||||
|
||||
## Contribution checklist
|
||||
|
||||
- Capture at 1920×1080 resolution unless otherwise specified.
|
||||
- Add annotations using the shared Docs Guild template (narrow callouts, numbered badges).
|
||||
- Optimize images to stay below 2 MB (PNG) and 8 MB (GIF) while preserving legibility.
|
||||
- Record GIFs at ≤30 seconds using 12–15 fps for balance between smoothness and size.
|
||||
- Update the capture checklist in `docs/examples/ui-tours.md` when assets are added or replaced.
|
||||
- Commit binaries using Git LFS if size exceeds repository limits; otherwise store directly.
|
||||
- Include the console build hash in the asset metadata or caption, matching the Downloads manifest version.
|
||||
# UI Tours Media Assets
|
||||
|
||||
Store annotated screenshots and GIFs referenced by `/docs/examples/ui-tours.md` in this directory. Use the naming convention documented in the guide (e.g., `triage-step-01.png`, `triage-flow.gif`).
|
||||
|
||||
## Contribution checklist
|
||||
|
||||
- Capture at 1920×1080 resolution unless otherwise specified.
|
||||
- Add annotations using the shared Docs Guild template (narrow callouts, numbered badges).
|
||||
- Optimize images to stay below 2 MB (PNG) and 8 MB (GIF) while preserving legibility.
|
||||
- Record GIFs at ≤30 seconds using 12–15 fps for balance between smoothness and size.
|
||||
- Update the capture checklist in `docs/examples/ui-tours.md` when assets are added or replaced.
|
||||
- Commit binaries using Git LFS if size exceeds repository limits; otherwise store directly.
|
||||
- Include the console build hash in the asset metadata or caption, matching the Downloads manifest version.
|
||||
|
||||
@@ -76,7 +76,7 @@ Use compressed JSON payloads, cached verification results, batched operations, a
|
||||
|
||||
## 4) Architecture
|
||||
|
||||
New services (`src/StellaOps.Attestor/`), libraries (`src/StellaOps.Attestor.Envelope/`, `src/StellaOps.Attestor.Types/`, `src/StellaOps.Attestor.Verify/`), CLI (`src/StellaOps.Cli/`), export tooling (`src/StellaOps.ExportCenter.AttestationBundles/`), and shared KMS providers (`src/StellaOps.Cryptography.Kms/`). REST endpoints documented in OpenAPI.
|
||||
New services (`src/Attestor/StellaOps.Attestor/`), libraries (`src/Attestor/StellaOps.Attestor.Envelope/`, `src/Attestor/StellaOps.Attestor.Types/`, `src/Attestor/StellaOps.Attestor.Verify/`), CLI (`src/Cli/StellaOps.Cli/`), export tooling (`src/ExportCenter/StellaOps.ExportCenter.AttestationBundles/`), and shared KMS providers (`src/__Libraries/StellaOps.Cryptography.Kms/`). REST endpoints documented in OpenAPI.
|
||||
|
||||
---
|
||||
|
||||
|
||||
@@ -12,6 +12,6 @@ This note captures the Sprint backlog hygiene pass applied on 26 October 2025. T
|
||||
- **CI/Offline adjustments.** `DEVOPS-UI-13-006` and `DEVOPS-OFFLINE-18-003` moved under Console release tasks (`CONSOLE-QA-23-401`, `DEVOPS-CONSOLE-23-001`, `CONSOLE-REL-23-302`).
|
||||
|
||||
## Follow-up
|
||||
- Update module task boards only under their active backlogs (`src/StellaOps.Notifier`, Cartographer, Vuln Explorer).
|
||||
- Update module task boards only under their active backlogs (`src/Notifier/StellaOps.Notifier`, Cartographer, Vuln Explorer).
|
||||
- Ensure future ingestion tasks reference AOC guardrails and avoid derived semantics.
|
||||
- Cross-check `SPRINTS.md` after adding new tasks to keep tables consistent with module `TASKS.md` files.
|
||||
- Cross-check `../implplan/SPRINTS.md` after adding new tasks to keep tables consistent with module `TASKS.md` files.
|
||||
|
||||
@@ -146,7 +146,7 @@ The script should emit a parity report that feeds into the Downloads workspace (
|
||||
- `/docs/install/docker.md` – CLI parity section for deployments.
|
||||
- `/docs/observability/ui-telemetry.md` – telemetry metrics referencing CLI checks.
|
||||
- `/docs/security/console-security.md` – security metrics & CLI parity expectations.
|
||||
- `src/StellaOps.Cli/TASKS.md` – authoritative status for CLI backlog.
|
||||
- `src/Cli/StellaOps.Cli/TASKS.md` – authoritative status for CLI backlog.
|
||||
- `/docs/updates/2025-10-28-docs-guild.md` – coordination note for Authority/Security follow-up.
|
||||
|
||||
---
|
||||
|
||||
@@ -1,316 +1,316 @@
|
||||
# CLI AOC Commands Reference
|
||||
|
||||
> **Audience:** DevEx engineers, operators, and CI authors integrating the `stella` CLI with Aggregation-Only Contract (AOC) workflows.
|
||||
> **Scope:** Command synopsis, options, exit codes, and offline considerations for `stella sources ingest --dry-run` and `stella aoc verify` as introduced in Sprint 19.
|
||||
|
||||
Both commands are designed to enforce the AOC guardrails documented in the [aggregation-only reference](../ingestion/aggregation-only-contract.md) and the [architecture overview](../architecture/overview.md). They consume Authority-issued tokens with tenant scopes and never mutate ingestion stores.
|
||||
|
||||
---
|
||||
|
||||
## 1 · Prerequisites
|
||||
|
||||
- CLI version: `stella` ≥ 0.19.0 (AOC feature gate enabled).
|
||||
- Required scopes (DPoP-bound):
|
||||
- `advisory:read` for Concelier sources.
|
||||
- `vex:read` for Excititor sources (optional but required for VEX checks).
|
||||
- `aoc:verify` to invoke guard verification endpoints.
|
||||
- `tenant:select` if your deployment uses tenant switching.
|
||||
- Connectivity: direct access to Concelier/Excititor APIs or Offline Kit snapshot (see § 4).
|
||||
- Environment: set `STELLA_AUTHORITY_URL`, `STELLA_TENANT`, and export a valid OpTok via `stella auth login` or existing token cache.
|
||||
|
||||
---
|
||||
|
||||
## 2 · `stella sources ingest --dry-run`
|
||||
|
||||
### 2.1 Synopsis
|
||||
|
||||
```bash
|
||||
stella sources ingest --dry-run \
|
||||
--source <source-key> \
|
||||
--input <path-or-uri> \
|
||||
[--tenant <tenant-id>] \
|
||||
[--format json|table] \
|
||||
[--no-color] \
|
||||
[--output <file>]
|
||||
```
|
||||
|
||||
### 2.2 Description
|
||||
|
||||
Previews an ingestion write without touching MongoDB. The command loads an upstream advisory or VEX document, computes the would-write payload, runs it through the `AOCWriteGuard`, and reports any forbidden fields, provenance gaps, or idempotency issues. Use it during connector development, CI validation, or while triaging incidents.
|
||||
|
||||
### 2.3 Options
|
||||
|
||||
| Option | Description |
|
||||
|--------|-------------|
|
||||
| `--source <source-key>` | Logical source name (`redhat`, `ubuntu`, `osv`, etc.). Mirrors connector configuration. |
|
||||
| `--input <path-or-uri>` | Path to local CSAF/OSV/VEX file or HTTPS URI. CLI normalises transport (gzip/base64) before guard evaluation. |
|
||||
| `--tenant <tenant-id>` | Overrides default tenant for multi-tenant deployments. Mandatory when `STELLA_TENANT` is not set. |
|
||||
| `--format json|table` | Output format. `table` (default) prints summary with highlighted violations; `json` emits machine-readable report (see below). |
|
||||
| `--no-color` | Disables ANSI colour output for CI logs. |
|
||||
| `--output <file>` | Writes the JSON report to file while still printing human-readable summary to stdout. |
|
||||
|
||||
### 2.4 Output schema (JSON)
|
||||
|
||||
```json
|
||||
{
|
||||
"source": "redhat",
|
||||
"tenant": "default",
|
||||
"guardVersion": "1.0.0",
|
||||
"status": "ok",
|
||||
"document": {
|
||||
"contentHash": "sha256:…",
|
||||
"supersedes": null,
|
||||
"provenance": {
|
||||
"signature": { "format": "pgp", "present": true }
|
||||
}
|
||||
},
|
||||
"violations": []
|
||||
}
|
||||
```
|
||||
|
||||
When violations exist, `status` becomes `error` and `violations` contains entries with `code` (`ERR_AOC_00x`), a short `message`, and JSON Pointer `path` values indicating offending fields.
|
||||
|
||||
### 2.5 Exit codes
|
||||
|
||||
| Exit code | Meaning |
|
||||
|-----------|---------|
|
||||
| `0` | Guard passed; would-write payload is AOC compliant. |
|
||||
| `11` | `ERR_AOC_001` – Forbidden field (`severity`, `cvss`, etc.) detected. |
|
||||
| `12` | `ERR_AOC_002` – Merge attempt (multiple upstream sources fused). |
|
||||
| `13` | `ERR_AOC_003` – Idempotency violation (duplicate without supersedes). |
|
||||
| `14` | `ERR_AOC_004` – Missing provenance fields. |
|
||||
| `15` | `ERR_AOC_005` – Signature/checksum mismatch. |
|
||||
| `16` | `ERR_AOC_006` – Effective findings present (Policy-only data). |
|
||||
| `17` | `ERR_AOC_007` – Unknown top-level fields / schema violation. |
|
||||
| `70` | Transport error (network, auth, malformed input). |
|
||||
|
||||
> Exit codes map directly to the `ERR_AOC_00x` table for scripting consistency. Multiple violations yield the highest-priority code (e.g., 11 takes precedence over 14).
|
||||
|
||||
### 2.6 Examples
|
||||
|
||||
Dry-run a local CSAF file:
|
||||
|
||||
```bash
|
||||
stella sources ingest --dry-run \
|
||||
--source redhat \
|
||||
--input ./fixtures/redhat/RHSA-2025-1234.json
|
||||
```
|
||||
|
||||
Stream from HTTPS and emit JSON for CI:
|
||||
|
||||
```bash
|
||||
stella sources ingest --dry-run \
|
||||
--source osv \
|
||||
--input https://osv.dev/vulnerability/GHSA-aaaa-bbbb \
|
||||
--format json \
|
||||
--output artifacts/osv-dry-run.json
|
||||
|
||||
cat artifacts/osv-dry-run.json | jq '.violations'
|
||||
```
|
||||
|
||||
### 2.7 Offline notes
|
||||
|
||||
When operating in sealed/offline mode:
|
||||
|
||||
- Use `--input` paths pointing to Offline Kit snapshots (`offline-kit/advisories/*.json`).
|
||||
- Provide `--tenant` explicitly if the offline bundle contains multiple tenants.
|
||||
- The command does not attempt network access when given a file path.
|
||||
- Store reports with `--output` to include in transfer packages for policy review.
|
||||
|
||||
---
|
||||
|
||||
## 3 · `stella aoc verify`
|
||||
|
||||
### 3.1 Synopsis
|
||||
|
||||
```bash
|
||||
stella aoc verify \
|
||||
[--since <iso8601|duration>] \
|
||||
[--limit <count>] \
|
||||
[--sources <list>] \
|
||||
[--codes <ERR_AOC_00x,...>] \
|
||||
[--format table|json] \
|
||||
[--export <file>] \
|
||||
[--tenant <tenant-id>] \
|
||||
[--no-color]
|
||||
```
|
||||
|
||||
### 3.2 Description
|
||||
|
||||
Replays the AOC guard against stored raw documents. By default it checks all advisories and VEX statements ingested in the last 24 hours for the active tenant, reporting totals, top violation codes, and sample documents. Use it in CI pipelines, scheduled verifications, or during incident response.
|
||||
|
||||
### 3.3 Options
|
||||
|
||||
| Option | Description |
|
||||
|--------|-------------|
|
||||
| `--since <value>` | Verification window. Accepts ISO 8601 timestamp (`2025-10-25T12:00:00Z`) or duration (`48h`, `7d`). Defaults to `24h`. |
|
||||
| `--limit <count>` | Maximum number of violations to display (per code). `0` means show all. Defaults to `20`. |
|
||||
| `--sources <list>` | Comma-separated list of sources (`redhat,ubuntu,osv`). Filters both advisories and VEX entries. |
|
||||
| `--codes <list>` | Restricts output to specific `ERR_AOC_00x` codes. Useful for regression tracking. |
|
||||
| `--format table|json` | `table` (default) prints summary plus top violations; `json` outputs machine-readable report identical to the `/aoc/verify` API. |
|
||||
| `--export <file>` | Writes the JSON report to disk (useful for audits/offline uploads). |
|
||||
| `--tenant <tenant-id>` | Overrides tenant context. Required for cross-tenant verifications when run by platform operators. |
|
||||
| `--no-color` | Disables ANSI colours. |
|
||||
|
||||
`table` mode prints a summary showing the active tenant, evaluated window, counts of checked advisories/VEX statements, the active limit, total writes/violations, and whether the page was truncated. Status is colour-coded as `ok`, `violations`, or `truncated`. When violations exist the detail table lists the code, total occurrences, first sample document (`source` + `documentId` + `contentHash`), and JSON pointer path.
|
||||
|
||||
### 3.4 Report structure (JSON)
|
||||
|
||||
```json
|
||||
{
|
||||
"tenant": "default",
|
||||
"window": {
|
||||
"from": "2025-10-25T12:00:00Z",
|
||||
"to": "2025-10-26T12:00:00Z"
|
||||
},
|
||||
"checked": {
|
||||
"advisories": 482,
|
||||
"vex": 75
|
||||
},
|
||||
"violations": [
|
||||
{
|
||||
"code": "ERR_AOC_001",
|
||||
"count": 2,
|
||||
"examples": [
|
||||
{
|
||||
"source": "redhat",
|
||||
"documentId": "advisory_raw:redhat:RHSA-2025:1",
|
||||
"contentHash": "sha256:…",
|
||||
"path": "/content/raw/cvss"
|
||||
}
|
||||
]
|
||||
}
|
||||
],
|
||||
"metrics": {
|
||||
"ingestion_write_total": 557,
|
||||
"aoc_violation_total": 2
|
||||
},
|
||||
"truncated": false
|
||||
}
|
||||
```
|
||||
|
||||
### 3.5 Exit codes
|
||||
|
||||
| Exit code | Meaning |
|
||||
|-----------|---------|
|
||||
| `0` | Verification succeeded with zero violations. |
|
||||
| `11…17` | Same mapping as § 2.5 when violations are detected. Highest-priority code returned. |
|
||||
| `18` | Verification ran but results truncated (limit reached) – treat as warning; rerun with higher `--limit`. |
|
||||
| `70` | Transport/authentication error. |
|
||||
| `71` | CLI misconfiguration (missing tenant, invalid `--since`, etc.). |
|
||||
|
||||
### 3.6 Examples
|
||||
|
||||
Daily verification across all sources:
|
||||
|
||||
```bash
|
||||
stella aoc verify --since 24h --format table
|
||||
```
|
||||
|
||||
CI pipeline focusing on errant sources and exporting evidence:
|
||||
|
||||
```bash
|
||||
stella aoc verify \
|
||||
--sources redhat,ubuntu \
|
||||
--codes ERR_AOC_001,ERR_AOC_004 \
|
||||
--format json \
|
||||
--limit 100 \
|
||||
--export artifacts/aoc-verify.json
|
||||
|
||||
jq '.violations[] | {code, count}' artifacts/aoc-verify.json
|
||||
```
|
||||
|
||||
Air-gapped verification using Offline Kit snapshot (example script):
|
||||
|
||||
```bash
|
||||
stella aoc verify \
|
||||
--since 7d \
|
||||
--format json \
|
||||
--export /mnt/offline/aoc-verify-$(date +%F).json
|
||||
|
||||
sha256sum /mnt/offline/aoc-verify-*.json > /mnt/offline/checksums.txt
|
||||
```
|
||||
|
||||
### 3.7 Automation tips
|
||||
|
||||
- Schedule with `cron` or platform scheduler and fail the job when exit code ≥ 11.
|
||||
- Pair with `stella sources ingest --dry-run` for pre-flight validation before re-enabling a paused source.
|
||||
- Push JSON exports to observability pipelines for historical tracking of violation counts.
|
||||
|
||||
### 3.8 Offline notes
|
||||
|
||||
- Works against Offline Kit Mongo snapshots when CLI is pointed at the local API gateway included in the bundle.
|
||||
- When fully disconnected, run against exported `aoc verify` reports generated on production and replay them using `--format json --export` (automation recipe above).
|
||||
- Include verification output in compliance packages alongside Offline Kit manifests.
|
||||
|
||||
---
|
||||
|
||||
## 4 · Global exit-code reference
|
||||
|
||||
| Code | Summary |
|
||||
|------|---------|
|
||||
| `0` | Success / no violations. |
|
||||
| `11` | `ERR_AOC_001` – Forbidden field present. |
|
||||
| `12` | `ERR_AOC_002` – Merge attempt detected. |
|
||||
| `13` | `ERR_AOC_003` – Idempotency violation. |
|
||||
| `14` | `ERR_AOC_004` – Missing provenance/signature metadata. |
|
||||
| `15` | `ERR_AOC_005` – Signature/checksum mismatch. |
|
||||
| `16` | `ERR_AOC_006` – Effective findings in ingestion payload. |
|
||||
| `17` | `ERR_AOC_007` – Schema violation / unknown fields. |
|
||||
| `18` | Partial verification (limit reached). |
|
||||
| `70` | Transport or HTTP failure. |
|
||||
| `71` | CLI usage error (invalid arguments, missing tenant). |
|
||||
|
||||
Use these codes in CI to map outcomes to build statuses or alert severities.
|
||||
|
||||
---
|
||||
|
||||
## 4 · `stella vuln observations` (Overlay paging)
|
||||
|
||||
`stella vuln observations` lists raw advisory observations for downstream overlays (Graph Explorer, Policy simulations, Console). Large tenants can now page through results deterministically.
|
||||
|
||||
| Option | Description |
|
||||
|--------|-------------|
|
||||
| `--limit <count>` | Caps the number of observations returned in a single call. Defaults to `200`; values above `500` are clamped server-side. |
|
||||
| `--cursor <token>` | Opaque continuation token produced by the previous page (`nextCursor` in JSON output). Pass it back to resume iteration. |
|
||||
|
||||
Additional notes:
|
||||
|
||||
- Table mode prints a hint when `hasMore` is `true`:
|
||||
`[yellow]More observations available. Continue with --cursor <token>[/]`.
|
||||
- JSON mode returns `nextCursor` and `hasMore` alongside the observation list so automation can loop until `hasMore` is `false`.
|
||||
- Supplying a non-positive limit falls back to the default (`200`). Invalid/expired cursors yield `400 Bad Request`; restart without `--cursor` to begin a fresh iteration.
|
||||
|
||||
---
|
||||
|
||||
## 5 · Related references
|
||||
|
||||
- [Aggregation-Only Contract reference](../ingestion/aggregation-only-contract.md)
|
||||
- [Architecture overview](../architecture/overview.md)
|
||||
- [Console AOC dashboard](../ui/console.md)
|
||||
- [Authority scopes](../ARCHITECTURE_AUTHORITY.md)
|
||||
|
||||
---
|
||||
|
||||
## 6 · Compliance checklist
|
||||
|
||||
- [ ] Usage documented for both table and JSON formats.
|
||||
- [ ] Exit-code mapping matches `ERR_AOC_00x` definitions and automation guidance.
|
||||
- [ ] Offline/air-gap workflow captured for both commands.
|
||||
- [ ] References to AOC architecture and console docs included.
|
||||
- [ ] Examples validated against current CLI syntax (update post-implementation).
|
||||
- [ ] Docs guild screenshot/narrative placeholder logged for release notes (pending CLI team capture).
|
||||
|
||||
---
|
||||
|
||||
*Last updated: 2025-10-29 (Sprint 24).*
|
||||
|
||||
## 13. Authority configuration quick reference
|
||||
|
||||
| Setting | Purpose | How to set |
|
||||
|---------|---------|------------|
|
||||
| `StellaOps:Authority:OperatorReason` | Incident/change description recorded with `orch:operate` tokens. | CLI flag `--Authority:OperatorReason=...` or env `STELLAOPS_ORCH_REASON`. |
|
||||
| `StellaOps:Authority:OperatorTicket` | Change/incident ticket reference paired with orchestrator control actions. | CLI flag `--Authority:OperatorTicket=...` or env `STELLAOPS_ORCH_TICKET`. |
|
||||
|
||||
> Tokens requesting `orch:operate` will fail with `invalid_request` unless both values are present. Choose concise strings (≤256 chars for reason, ≤128 chars for ticket) and avoid sensitive data.
|
||||
|
||||
# CLI AOC Commands Reference
|
||||
|
||||
> **Audience:** DevEx engineers, operators, and CI authors integrating the `stella` CLI with Aggregation-Only Contract (AOC) workflows.
|
||||
> **Scope:** Command synopsis, options, exit codes, and offline considerations for `stella sources ingest --dry-run` and `stella aoc verify` as introduced in Sprint 19.
|
||||
|
||||
Both commands are designed to enforce the AOC guardrails documented in the [aggregation-only reference](../ingestion/aggregation-only-contract.md) and the [architecture overview](../architecture/overview.md). They consume Authority-issued tokens with tenant scopes and never mutate ingestion stores.
|
||||
|
||||
---
|
||||
|
||||
## 1 · Prerequisites
|
||||
|
||||
- CLI version: `stella` ≥ 0.19.0 (AOC feature gate enabled).
|
||||
- Required scopes (DPoP-bound):
|
||||
- `advisory:read` for Concelier sources.
|
||||
- `vex:read` for Excititor sources (optional but required for VEX checks).
|
||||
- `aoc:verify` to invoke guard verification endpoints.
|
||||
- `tenant:select` if your deployment uses tenant switching.
|
||||
- Connectivity: direct access to Concelier/Excititor APIs or Offline Kit snapshot (see § 4).
|
||||
- Environment: set `STELLA_AUTHORITY_URL`, `STELLA_TENANT`, and export a valid OpTok via `stella auth login` or existing token cache.
|
||||
|
||||
---
|
||||
|
||||
## 2 · `stella sources ingest --dry-run`
|
||||
|
||||
### 2.1 Synopsis
|
||||
|
||||
```bash
|
||||
stella sources ingest --dry-run \
|
||||
--source <source-key> \
|
||||
--input <path-or-uri> \
|
||||
[--tenant <tenant-id>] \
|
||||
[--format json|table] \
|
||||
[--no-color] \
|
||||
[--output <file>]
|
||||
```
|
||||
|
||||
### 2.2 Description
|
||||
|
||||
Previews an ingestion write without touching MongoDB. The command loads an upstream advisory or VEX document, computes the would-write payload, runs it through the `AOCWriteGuard`, and reports any forbidden fields, provenance gaps, or idempotency issues. Use it during connector development, CI validation, or while triaging incidents.
|
||||
|
||||
### 2.3 Options
|
||||
|
||||
| Option | Description |
|
||||
|--------|-------------|
|
||||
| `--source <source-key>` | Logical source name (`redhat`, `ubuntu`, `osv`, etc.). Mirrors connector configuration. |
|
||||
| `--input <path-or-uri>` | Path to local CSAF/OSV/VEX file or HTTPS URI. CLI normalises transport (gzip/base64) before guard evaluation. |
|
||||
| `--tenant <tenant-id>` | Overrides default tenant for multi-tenant deployments. Mandatory when `STELLA_TENANT` is not set. |
|
||||
| `--format json|table` | Output format. `table` (default) prints summary with highlighted violations; `json` emits machine-readable report (see below). |
|
||||
| `--no-color` | Disables ANSI colour output for CI logs. |
|
||||
| `--output <file>` | Writes the JSON report to file while still printing human-readable summary to stdout. |
|
||||
|
||||
### 2.4 Output schema (JSON)
|
||||
|
||||
```json
|
||||
{
|
||||
"source": "redhat",
|
||||
"tenant": "default",
|
||||
"guardVersion": "1.0.0",
|
||||
"status": "ok",
|
||||
"document": {
|
||||
"contentHash": "sha256:…",
|
||||
"supersedes": null,
|
||||
"provenance": {
|
||||
"signature": { "format": "pgp", "present": true }
|
||||
}
|
||||
},
|
||||
"violations": []
|
||||
}
|
||||
```
|
||||
|
||||
When violations exist, `status` becomes `error` and `violations` contains entries with `code` (`ERR_AOC_00x`), a short `message`, and JSON Pointer `path` values indicating offending fields.
|
||||
|
||||
### 2.5 Exit codes
|
||||
|
||||
| Exit code | Meaning |
|
||||
|-----------|---------|
|
||||
| `0` | Guard passed; would-write payload is AOC compliant. |
|
||||
| `11` | `ERR_AOC_001` – Forbidden field (`severity`, `cvss`, etc.) detected. |
|
||||
| `12` | `ERR_AOC_002` – Merge attempt (multiple upstream sources fused). |
|
||||
| `13` | `ERR_AOC_003` – Idempotency violation (duplicate without supersedes). |
|
||||
| `14` | `ERR_AOC_004` – Missing provenance fields. |
|
||||
| `15` | `ERR_AOC_005` – Signature/checksum mismatch. |
|
||||
| `16` | `ERR_AOC_006` – Effective findings present (Policy-only data). |
|
||||
| `17` | `ERR_AOC_007` – Unknown top-level fields / schema violation. |
|
||||
| `70` | Transport error (network, auth, malformed input). |
|
||||
|
||||
> Exit codes map directly to the `ERR_AOC_00x` table for scripting consistency. Multiple violations yield the highest-priority code (e.g., 11 takes precedence over 14).
|
||||
|
||||
### 2.6 Examples
|
||||
|
||||
Dry-run a local CSAF file:
|
||||
|
||||
```bash
|
||||
stella sources ingest --dry-run \
|
||||
--source redhat \
|
||||
--input ./fixtures/redhat/RHSA-2025-1234.json
|
||||
```
|
||||
|
||||
Stream from HTTPS and emit JSON for CI:
|
||||
|
||||
```bash
|
||||
stella sources ingest --dry-run \
|
||||
--source osv \
|
||||
--input https://osv.dev/vulnerability/GHSA-aaaa-bbbb \
|
||||
--format json \
|
||||
--output artifacts/osv-dry-run.json
|
||||
|
||||
cat artifacts/osv-dry-run.json | jq '.violations'
|
||||
```
|
||||
|
||||
### 2.7 Offline notes
|
||||
|
||||
When operating in sealed/offline mode:
|
||||
|
||||
- Use `--input` paths pointing to Offline Kit snapshots (`offline-kit/advisories/*.json`).
|
||||
- Provide `--tenant` explicitly if the offline bundle contains multiple tenants.
|
||||
- The command does not attempt network access when given a file path.
|
||||
- Store reports with `--output` to include in transfer packages for policy review.
|
||||
|
||||
---
|
||||
|
||||
## 3 · `stella aoc verify`
|
||||
|
||||
### 3.1 Synopsis
|
||||
|
||||
```bash
|
||||
stella aoc verify \
|
||||
[--since <iso8601|duration>] \
|
||||
[--limit <count>] \
|
||||
[--sources <list>] \
|
||||
[--codes <ERR_AOC_00x,...>] \
|
||||
[--format table|json] \
|
||||
[--export <file>] \
|
||||
[--tenant <tenant-id>] \
|
||||
[--no-color]
|
||||
```
|
||||
|
||||
### 3.2 Description
|
||||
|
||||
Replays the AOC guard against stored raw documents. By default it checks all advisories and VEX statements ingested in the last 24 hours for the active tenant, reporting totals, top violation codes, and sample documents. Use it in CI pipelines, scheduled verifications, or during incident response.
|
||||
|
||||
### 3.3 Options
|
||||
|
||||
| Option | Description |
|
||||
|--------|-------------|
|
||||
| `--since <value>` | Verification window. Accepts ISO 8601 timestamp (`2025-10-25T12:00:00Z`) or duration (`48h`, `7d`). Defaults to `24h`. |
|
||||
| `--limit <count>` | Maximum number of violations to display (per code). `0` means show all. Defaults to `20`. |
|
||||
| `--sources <list>` | Comma-separated list of sources (`redhat,ubuntu,osv`). Filters both advisories and VEX entries. |
|
||||
| `--codes <list>` | Restricts output to specific `ERR_AOC_00x` codes. Useful for regression tracking. |
|
||||
| `--format table|json` | `table` (default) prints summary plus top violations; `json` outputs machine-readable report identical to the `/aoc/verify` API. |
|
||||
| `--export <file>` | Writes the JSON report to disk (useful for audits/offline uploads). |
|
||||
| `--tenant <tenant-id>` | Overrides tenant context. Required for cross-tenant verifications when run by platform operators. |
|
||||
| `--no-color` | Disables ANSI colours. |
|
||||
|
||||
`table` mode prints a summary showing the active tenant, evaluated window, counts of checked advisories/VEX statements, the active limit, total writes/violations, and whether the page was truncated. Status is colour-coded as `ok`, `violations`, or `truncated`. When violations exist the detail table lists the code, total occurrences, first sample document (`source` + `documentId` + `contentHash`), and JSON pointer path.
|
||||
|
||||
### 3.4 Report structure (JSON)
|
||||
|
||||
```json
|
||||
{
|
||||
"tenant": "default",
|
||||
"window": {
|
||||
"from": "2025-10-25T12:00:00Z",
|
||||
"to": "2025-10-26T12:00:00Z"
|
||||
},
|
||||
"checked": {
|
||||
"advisories": 482,
|
||||
"vex": 75
|
||||
},
|
||||
"violations": [
|
||||
{
|
||||
"code": "ERR_AOC_001",
|
||||
"count": 2,
|
||||
"examples": [
|
||||
{
|
||||
"source": "redhat",
|
||||
"documentId": "advisory_raw:redhat:RHSA-2025:1",
|
||||
"contentHash": "sha256:…",
|
||||
"path": "/content/raw/cvss"
|
||||
}
|
||||
]
|
||||
}
|
||||
],
|
||||
"metrics": {
|
||||
"ingestion_write_total": 557,
|
||||
"aoc_violation_total": 2
|
||||
},
|
||||
"truncated": false
|
||||
}
|
||||
```
|
||||
|
||||
### 3.5 Exit codes
|
||||
|
||||
| Exit code | Meaning |
|
||||
|-----------|---------|
|
||||
| `0` | Verification succeeded with zero violations. |
|
||||
| `11…17` | Same mapping as § 2.5 when violations are detected. Highest-priority code returned. |
|
||||
| `18` | Verification ran but results truncated (limit reached) – treat as warning; rerun with higher `--limit`. |
|
||||
| `70` | Transport/authentication error. |
|
||||
| `71` | CLI misconfiguration (missing tenant, invalid `--since`, etc.). |
|
||||
|
||||
### 3.6 Examples
|
||||
|
||||
Daily verification across all sources:
|
||||
|
||||
```bash
|
||||
stella aoc verify --since 24h --format table
|
||||
```
|
||||
|
||||
CI pipeline focusing on errant sources and exporting evidence:
|
||||
|
||||
```bash
|
||||
stella aoc verify \
|
||||
--sources redhat,ubuntu \
|
||||
--codes ERR_AOC_001,ERR_AOC_004 \
|
||||
--format json \
|
||||
--limit 100 \
|
||||
--export artifacts/aoc-verify.json
|
||||
|
||||
jq '.violations[] | {code, count}' artifacts/aoc-verify.json
|
||||
```
|
||||
|
||||
Air-gapped verification using Offline Kit snapshot (example script):
|
||||
|
||||
```bash
|
||||
stella aoc verify \
|
||||
--since 7d \
|
||||
--format json \
|
||||
--export /mnt/offline/aoc-verify-$(date +%F).json
|
||||
|
||||
sha256sum /mnt/offline/aoc-verify-*.json > /mnt/offline/checksums.txt
|
||||
```
|
||||
|
||||
### 3.7 Automation tips
|
||||
|
||||
- Schedule with `cron` or platform scheduler and fail the job when exit code ≥ 11.
|
||||
- Pair with `stella sources ingest --dry-run` for pre-flight validation before re-enabling a paused source.
|
||||
- Push JSON exports to observability pipelines for historical tracking of violation counts.
|
||||
|
||||
### 3.8 Offline notes
|
||||
|
||||
- Works against Offline Kit Mongo snapshots when CLI is pointed at the local API gateway included in the bundle.
|
||||
- When fully disconnected, run against exported `aoc verify` reports generated on production and replay them using `--format json --export` (automation recipe above).
|
||||
- Include verification output in compliance packages alongside Offline Kit manifests.
|
||||
|
||||
---
|
||||
|
||||
## 4 · Global exit-code reference
|
||||
|
||||
| Code | Summary |
|
||||
|------|---------|
|
||||
| `0` | Success / no violations. |
|
||||
| `11` | `ERR_AOC_001` – Forbidden field present. |
|
||||
| `12` | `ERR_AOC_002` – Merge attempt detected. |
|
||||
| `13` | `ERR_AOC_003` – Idempotency violation. |
|
||||
| `14` | `ERR_AOC_004` – Missing provenance/signature metadata. |
|
||||
| `15` | `ERR_AOC_005` – Signature/checksum mismatch. |
|
||||
| `16` | `ERR_AOC_006` – Effective findings in ingestion payload. |
|
||||
| `17` | `ERR_AOC_007` – Schema violation / unknown fields. |
|
||||
| `18` | Partial verification (limit reached). |
|
||||
| `70` | Transport or HTTP failure. |
|
||||
| `71` | CLI usage error (invalid arguments, missing tenant). |
|
||||
|
||||
Use these codes in CI to map outcomes to build statuses or alert severities.
|
||||
|
||||
---
|
||||
|
||||
## 4 · `stella vuln observations` (Overlay paging)
|
||||
|
||||
`stella vuln observations` lists raw advisory observations for downstream overlays (Graph Explorer, Policy simulations, Console). Large tenants can now page through results deterministically.
|
||||
|
||||
| Option | Description |
|
||||
|--------|-------------|
|
||||
| `--limit <count>` | Caps the number of observations returned in a single call. Defaults to `200`; values above `500` are clamped server-side. |
|
||||
| `--cursor <token>` | Opaque continuation token produced by the previous page (`nextCursor` in JSON output). Pass it back to resume iteration. |
|
||||
|
||||
Additional notes:
|
||||
|
||||
- Table mode prints a hint when `hasMore` is `true`:
|
||||
`[yellow]More observations available. Continue with --cursor <token>[/]`.
|
||||
- JSON mode returns `nextCursor` and `hasMore` alongside the observation list so automation can loop until `hasMore` is `false`.
|
||||
- Supplying a non-positive limit falls back to the default (`200`). Invalid/expired cursors yield `400 Bad Request`; restart without `--cursor` to begin a fresh iteration.
|
||||
|
||||
---
|
||||
|
||||
## 5 · Related references
|
||||
|
||||
- [Aggregation-Only Contract reference](../ingestion/aggregation-only-contract.md)
|
||||
- [Architecture overview](../architecture/overview.md)
|
||||
- [Console AOC dashboard](../ui/console.md)
|
||||
- [Authority scopes](../ARCHITECTURE_AUTHORITY.md)
|
||||
|
||||
---
|
||||
|
||||
## 6 · Compliance checklist
|
||||
|
||||
- [ ] Usage documented for both table and JSON formats.
|
||||
- [ ] Exit-code mapping matches `ERR_AOC_00x` definitions and automation guidance.
|
||||
- [ ] Offline/air-gap workflow captured for both commands.
|
||||
- [ ] References to AOC architecture and console docs included.
|
||||
- [ ] Examples validated against current CLI syntax (update post-implementation).
|
||||
- [ ] Docs guild screenshot/narrative placeholder logged for release notes (pending CLI team capture).
|
||||
|
||||
---
|
||||
|
||||
*Last updated: 2025-10-29 (Sprint 24).*
|
||||
|
||||
## 13. Authority configuration quick reference
|
||||
|
||||
| Setting | Purpose | How to set |
|
||||
|---------|---------|------------|
|
||||
| `StellaOps:Authority:OperatorReason` | Incident/change description recorded with `orch:operate` tokens. | CLI flag `--Authority:OperatorReason=...` or env `STELLAOPS_ORCH_REASON`. |
|
||||
| `StellaOps:Authority:OperatorTicket` | Change/incident ticket reference paired with orchestrator control actions. | CLI flag `--Authority:OperatorTicket=...` or env `STELLAOPS_ORCH_TICKET`. |
|
||||
|
||||
> Tokens requesting `orch:operate` will fail with `invalid_request` unless both values are present. Choose concise strings (≤256 chars for reason, ≤128 chars for ticket) and avoid sensitive data.
|
||||
|
||||
|
||||
@@ -185,7 +185,7 @@ Output fields (JSON):
|
||||
}
|
||||
```
|
||||
|
||||
> Schema reminder: CLI commands surface objects defined in `src/StellaOps.Scheduler.Models/docs/SCHED-MODELS-20-001-POLICY-RUNS.md`; use the samples in `samples/api/scheduler/` for contract validation when extending output parsing.
|
||||
> Schema reminder: CLI commands surface objects defined in `src/Scheduler/__Libraries/StellaOps.Scheduler.Models/docs/SCHED-MODELS-20-001-POLICY-RUNS.md`; use the samples in `samples/api/scheduler/` for contract validation when extending output parsing.
|
||||
|
||||
Exit codes:
|
||||
|
||||
|
||||
@@ -1,228 +1,228 @@
|
||||
# Deploying the StellaOps Console
|
||||
|
||||
> **Audience:** Deployment Guild, Console Guild, operators rolling out the web console.
|
||||
> **Scope:** Helm and Docker Compose deployment steps, ingress/TLS configuration, required environment variables, health checks, offline/air-gap operation, and compliance checklist (Sprint 23).
|
||||
|
||||
The StellaOps Console ships as part of the `stellaops` stack Helm chart and Compose bundles maintained under `deploy/`. This guide describes the supported deployment paths, the configuration surface, and operational checks needed to run the console in connected or air-gapped environments.
|
||||
|
||||
---
|
||||
|
||||
## 1. Prerequisites
|
||||
|
||||
- Kubernetes cluster (v1.28+) with ingress controller (NGINX, Traefik, or equivalent) and Cert-Manager for automated TLS, or Docker host for Compose deployments.
|
||||
- Container registry access to `registry.stella-ops.org` (or mirrored registry) for all images listed in `deploy/releases/*.yaml`.
|
||||
- Authority service configured with console client (`aud=ui`, scopes `ui.read`, `ui.admin`).
|
||||
- DNS entry pointing to the console hostname (for example, `console.acme.internal`).
|
||||
- Cosign public key for manifest verification (`deploy/releases/manifest.json.sig`).
|
||||
- Optional: Offline Kit bundle for air-gapped sites (`stella-ops-offline-kit-<ver>.tar.gz`).
|
||||
|
||||
---
|
||||
|
||||
## 2. Helm deployment (recommended)
|
||||
|
||||
### 2.1 Install chart repository
|
||||
|
||||
```bash
|
||||
helm repo add stellaops https://downloads.stella-ops.org/helm
|
||||
helm repo update stellaops
|
||||
```
|
||||
|
||||
If operating offline, copy the chart archive from the Offline Kit (`deploy/helm/stellaops-<ver>.tgz`) and run:
|
||||
|
||||
```bash
|
||||
helm install stellaops ./stellaops-<ver>.tgz --namespace stellaops --create-namespace
|
||||
```
|
||||
|
||||
### 2.2 Base installation
|
||||
|
||||
```bash
|
||||
helm install stellaops stellaops/stellaops \
|
||||
--namespace stellaops \
|
||||
--create-namespace \
|
||||
--values deploy/helm/stellaops/values-prod.yaml
|
||||
```
|
||||
|
||||
The chart deploys Authority, Console web/API gateway, Scanner API, Scheduler, and supporting services. The console frontend pod is labelled `app=stellaops-web-ui`.
|
||||
|
||||
### 2.3 Helm values highlights
|
||||
|
||||
Key sections in `deploy/helm/stellaops/values-prod.yaml`:
|
||||
|
||||
| Path | Description |
|
||||
|------|-------------|
|
||||
| `console.ingress.host` | Hostname served by the console (`console.example.com`). |
|
||||
| `console.ingress.tls.secretName` | Kubernetes secret containing TLS certificate (generated by Cert-Manager or uploaded manually). |
|
||||
| `console.config.apiGateway.baseUrl` | Internal base URL the UI uses to reach the gateway (defaults to `https://stellaops-web`). |
|
||||
| `console.env.AUTHORITY_ISSUER` | Authority issuer URL (for example, `https://authority.example.com`). |
|
||||
| `console.env.AUTHORITY_CLIENT_ID` | Authority client ID for the console UI. |
|
||||
| `console.env.AUTHORITY_SCOPES` | Space-separated scopes required by UI (`ui.read ui.admin`). |
|
||||
| `console.resources` | CPU/memory requests and limits (default 250m CPU / 512Mi memory). |
|
||||
| `console.podAnnotations` | Optional annotations for service mesh or monitoring. |
|
||||
|
||||
Use `values-stage.yaml`, `values-dev.yaml`, or `values-airgap.yaml` as templates for other environments.
|
||||
|
||||
### 2.4 TLS and ingress
|
||||
|
||||
Example ingress override:
|
||||
|
||||
```yaml
|
||||
console:
|
||||
ingress:
|
||||
enabled: true
|
||||
className: nginx
|
||||
host: console.acme.internal
|
||||
tls:
|
||||
enabled: true
|
||||
secretName: console-tls
|
||||
```
|
||||
|
||||
Generate certificates using Cert-Manager or provide an existing secret. For air-gapped deployments, pre-create the secret with the mirrored CA chain.
|
||||
|
||||
### 2.5 Health checks
|
||||
|
||||
Console pods expose:
|
||||
|
||||
| Path | Purpose | Notes |
|
||||
|------|---------|-------|
|
||||
| `/health/live` | Liveness probe | Confirms process responsive. |
|
||||
| `/health/ready` | Readiness probe | Verifies configuration bootstrap and Authority reachability. |
|
||||
| `/metrics` | Prometheus metrics | Enabled when `console.metrics.enabled=true`. |
|
||||
|
||||
Helm chart sets default probes (`initialDelaySeconds: 10`, `periodSeconds: 15`). Adjust via `console.livenessProbe` and `console.readinessProbe`.
|
||||
|
||||
---
|
||||
|
||||
## 3. Docker Compose deployment
|
||||
|
||||
Located in `deploy/compose/docker-compose.console.yaml`. Quick start:
|
||||
|
||||
```bash
|
||||
cd deploy/compose
|
||||
docker compose -f docker-compose.console.yaml --env-file console.env up -d
|
||||
```
|
||||
|
||||
`console.env` should define:
|
||||
|
||||
```
|
||||
CONSOLE_PUBLIC_BASE_URL=https://console.acme.internal
|
||||
AUTHORITY_ISSUER=https://authority.acme.internal
|
||||
AUTHORITY_CLIENT_ID=console-ui
|
||||
AUTHORITY_CLIENT_SECRET=<if using confidential client>
|
||||
AUTHORITY_SCOPES=ui.read ui.admin
|
||||
CONSOLE_GATEWAY_BASE_URL=https://api.acme.internal
|
||||
```
|
||||
|
||||
The compose bundle includes Traefik as reverse proxy with TLS termination. Update `traefik/dynamic/console.yml` for custom certificates or additional middlewares (CSP headers, rate limits).
|
||||
|
||||
---
|
||||
|
||||
## 4. Environment variables
|
||||
|
||||
| Variable | Description | Default |
|
||||
|----------|-------------|---------|
|
||||
| `CONSOLE_PUBLIC_BASE_URL` | External URL used for redirects, deep links, and telemetry. | None (required). |
|
||||
| `CONSOLE_GATEWAY_BASE_URL` | URL of the web gateway that proxies API calls (`/console/*`). | Chart service name. |
|
||||
| `AUTHORITY_ISSUER` | Authority issuer (`https://authority.example.com`). | None (required). |
|
||||
| `AUTHORITY_CLIENT_ID` | OIDC client configured in Authority. | None (required). |
|
||||
| `AUTHORITY_SCOPES` | Space-separated scopes assigned to the console client. | `ui.read ui.admin`. |
|
||||
| `AUTHORITY_DPOP_ENABLED` | Enables DPoP challenge/response (recommended true). | `true`. |
|
||||
| `CONSOLE_FEATURE_FLAGS` | Comma-separated feature flags (`runs`, `downloads.offline`, etc.). | `runs,downloads,policies`. |
|
||||
| `CONSOLE_LOG_LEVEL` | Minimum log level (`Information`, `Debug`, etc.). | `Information`. |
|
||||
| `CONSOLE_METRICS_ENABLED` | Expose `/metrics` endpoint. | `true`. |
|
||||
| `CONSOLE_SENTRY_DSN` | Optional error reporting DSN. | Blank. |
|
||||
|
||||
When running behind additional proxies, set `ASPNETCORE_FORWARDEDHEADERS_ENABLED=true` to honour `X-Forwarded-*` headers.
|
||||
|
||||
---
|
||||
|
||||
## 5. Security headers and CSP
|
||||
|
||||
The console serves a strict Content Security Policy (CSP) by default:
|
||||
|
||||
```
|
||||
default-src 'self';
|
||||
connect-src 'self' https://*.stella-ops.local;
|
||||
script-src 'self';
|
||||
style-src 'self' 'unsafe-inline';
|
||||
img-src 'self' data:;
|
||||
font-src 'self';
|
||||
frame-ancestors 'none';
|
||||
```
|
||||
|
||||
Adjust via `console.config.cspOverrides` if additional domains are required. For integrations embedding the console, update OIDC redirect URIs and Authority scopes accordingly.
|
||||
|
||||
TLS recommendations:
|
||||
|
||||
- Use TLS 1.2+ with modern cipher suite policy.
|
||||
- Enable HSTS (`Strict-Transport-Security: max-age=31536000; includeSubDomains`).
|
||||
- Provide custom trust bundles via `console.config.trustBundleSecret` when using private CAs.
|
||||
|
||||
---
|
||||
|
||||
## 6. Logging and metrics
|
||||
|
||||
- Structured logs emitted to stdout with correlation IDs. Configure log shipping via Fluent Bit or similar.
|
||||
- Metrics available at `/metrics` in Prometheus format. Key metrics include `ui_request_duration_seconds`, `ui_tenant_switch_total`, and `ui_download_manifest_refresh_seconds`.
|
||||
- Enable OpenTelemetry exporter by setting `OTEL_EXPORTER_OTLP_ENDPOINT` and associated headers in environment variables.
|
||||
|
||||
---
|
||||
|
||||
## 7. Offline and air-gap deployment
|
||||
|
||||
- Mirror container images using the Downloads workspace or Offline Kit manifest. Example:
|
||||
|
||||
```bash
|
||||
oras copy registry.stella-ops.org/stellaops/web-ui@sha256:<digest> \
|
||||
registry.airgap.local/stellaops/web-ui:2025.10.0
|
||||
```
|
||||
|
||||
- Import Offline Kit using `stella ouk import` before starting the console so manifest parity checks succeed.
|
||||
- Use `values-airgap.yaml` to disable external telemetry endpoints and configure internal certificate chains.
|
||||
- Run `helm upgrade --install` using the mirrored chart (`stellaops-<ver>.tgz`) and set `console.offlineMode=true` to surface offline banners.
|
||||
|
||||
---
|
||||
|
||||
## 8. Health checks and remediation
|
||||
|
||||
| Check | Command | Expected result |
|
||||
|-------|---------|-----------------|
|
||||
| Pod status | `kubectl get pods -n stellaops` | `Running` state with restarts = 0. |
|
||||
| Liveness | `kubectl exec deploy/stellaops-web-ui -- curl -fsS http://localhost:8080/health/live` | Returns `{"status":"Healthy"}`. |
|
||||
| Readiness | `kubectl exec deploy/stellaops-web-ui -- curl -fsS http://localhost:8080/health/ready` | Returns `{"status":"Ready"}`. |
|
||||
| Gateway reachability | `curl -I https://console.example.com/api/console/status` | `200 OK` with CSP headers. |
|
||||
| Static assets | `curl -I https://console.example.com/static/assets/app.js` | `200 OK` with long cache headers. |
|
||||
|
||||
Troubleshooting steps:
|
||||
|
||||
- **Authority unreachable:** readiness fails with `AUTHORITY_UNREACHABLE`. Check DNS, trust bundles, and Authority service health.
|
||||
- **Manifest mismatch:** console logs `DOWNLOAD_MANIFEST_SIGNATURE_INVALID`. Verify cosign key and re-sync manifest.
|
||||
- **Ingress 404:** ensure ingress controller routes host to `stellaops-web-ui` service; check TLS secret name.
|
||||
- **SSE blocked:** confirm proxy allows HTTP/1.1 and disables buffering on `/console/runs/*`.
|
||||
|
||||
---
|
||||
|
||||
## 9. References
|
||||
|
||||
- `deploy/helm/stellaops/values-*.yaml` - environment-specific overrides.
|
||||
- `deploy/compose/docker-compose.console.yaml` - Compose bundle.
|
||||
- `/docs/ui/downloads.md` - manifest and offline bundle guidance.
|
||||
- `/docs/security/console-security.md` - CSP and Authority scopes.
|
||||
- `/docs/24_OFFLINE_KIT.md` - Offline kit packaging and verification.
|
||||
- `/docs/ops/deployment-runbook.md` (pending) - wider platform deployment steps.
|
||||
|
||||
---
|
||||
|
||||
## 10. Compliance checklist
|
||||
|
||||
- [ ] Helm and Compose instructions verified against `deploy/` assets.
|
||||
- [ ] Ingress/TLS guidance aligns with Security Guild recommendations.
|
||||
- [ ] Environment variables documented with defaults and required values.
|
||||
- [ ] Health/liveness/readiness endpoints tested and listed.
|
||||
- [ ] Offline workflow (mirrors, manifest parity) captured.
|
||||
- [ ] Logging and metrics surface documented metrics.
|
||||
- [ ] CSP and security header defaults stated alongside override guidance.
|
||||
- [ ] Troubleshooting section linked to relevant runbooks.
|
||||
|
||||
---
|
||||
|
||||
*Last updated: 2025-10-27 (Sprint 23).*
|
||||
# Deploying the StellaOps Console
|
||||
|
||||
> **Audience:** Deployment Guild, Console Guild, operators rolling out the web console.
|
||||
> **Scope:** Helm and Docker Compose deployment steps, ingress/TLS configuration, required environment variables, health checks, offline/air-gap operation, and compliance checklist (Sprint 23).
|
||||
|
||||
The StellaOps Console ships as part of the `stellaops` stack Helm chart and Compose bundles maintained under `deploy/`. This guide describes the supported deployment paths, the configuration surface, and operational checks needed to run the console in connected or air-gapped environments.
|
||||
|
||||
---
|
||||
|
||||
## 1. Prerequisites
|
||||
|
||||
- Kubernetes cluster (v1.28+) with ingress controller (NGINX, Traefik, or equivalent) and Cert-Manager for automated TLS, or Docker host for Compose deployments.
|
||||
- Container registry access to `registry.stella-ops.org` (or mirrored registry) for all images listed in `deploy/releases/*.yaml`.
|
||||
- Authority service configured with console client (`aud=ui`, scopes `ui.read`, `ui.admin`).
|
||||
- DNS entry pointing to the console hostname (for example, `console.acme.internal`).
|
||||
- Cosign public key for manifest verification (`deploy/releases/manifest.json.sig`).
|
||||
- Optional: Offline Kit bundle for air-gapped sites (`stella-ops-offline-kit-<ver>.tar.gz`).
|
||||
|
||||
---
|
||||
|
||||
## 2. Helm deployment (recommended)
|
||||
|
||||
### 2.1 Install chart repository
|
||||
|
||||
```bash
|
||||
helm repo add stellaops https://downloads.stella-ops.org/helm
|
||||
helm repo update stellaops
|
||||
```
|
||||
|
||||
If operating offline, copy the chart archive from the Offline Kit (`deploy/helm/stellaops-<ver>.tgz`) and run:
|
||||
|
||||
```bash
|
||||
helm install stellaops ./stellaops-<ver>.tgz --namespace stellaops --create-namespace
|
||||
```
|
||||
|
||||
### 2.2 Base installation
|
||||
|
||||
```bash
|
||||
helm install stellaops stellaops/stellaops \
|
||||
--namespace stellaops \
|
||||
--create-namespace \
|
||||
--values deploy/helm/stellaops/values-prod.yaml
|
||||
```
|
||||
|
||||
The chart deploys Authority, Console web/API gateway, Scanner API, Scheduler, and supporting services. The console frontend pod is labelled `app=stellaops-web-ui`.
|
||||
|
||||
### 2.3 Helm values highlights
|
||||
|
||||
Key sections in `deploy/helm/stellaops/values-prod.yaml`:
|
||||
|
||||
| Path | Description |
|
||||
|------|-------------|
|
||||
| `console.ingress.host` | Hostname served by the console (`console.example.com`). |
|
||||
| `console.ingress.tls.secretName` | Kubernetes secret containing TLS certificate (generated by Cert-Manager or uploaded manually). |
|
||||
| `console.config.apiGateway.baseUrl` | Internal base URL the UI uses to reach the gateway (defaults to `https://stellaops-web`). |
|
||||
| `console.env.AUTHORITY_ISSUER` | Authority issuer URL (for example, `https://authority.example.com`). |
|
||||
| `console.env.AUTHORITY_CLIENT_ID` | Authority client ID for the console UI. |
|
||||
| `console.env.AUTHORITY_SCOPES` | Space-separated scopes required by UI (`ui.read ui.admin`). |
|
||||
| `console.resources` | CPU/memory requests and limits (default 250m CPU / 512Mi memory). |
|
||||
| `console.podAnnotations` | Optional annotations for service mesh or monitoring. |
|
||||
|
||||
Use `values-stage.yaml`, `values-dev.yaml`, or `values-airgap.yaml` as templates for other environments.
|
||||
|
||||
### 2.4 TLS and ingress
|
||||
|
||||
Example ingress override:
|
||||
|
||||
```yaml
|
||||
console:
|
||||
ingress:
|
||||
enabled: true
|
||||
className: nginx
|
||||
host: console.acme.internal
|
||||
tls:
|
||||
enabled: true
|
||||
secretName: console-tls
|
||||
```
|
||||
|
||||
Generate certificates using Cert-Manager or provide an existing secret. For air-gapped deployments, pre-create the secret with the mirrored CA chain.
|
||||
|
||||
### 2.5 Health checks
|
||||
|
||||
Console pods expose:
|
||||
|
||||
| Path | Purpose | Notes |
|
||||
|------|---------|-------|
|
||||
| `/health/live` | Liveness probe | Confirms process responsive. |
|
||||
| `/health/ready` | Readiness probe | Verifies configuration bootstrap and Authority reachability. |
|
||||
| `/metrics` | Prometheus metrics | Enabled when `console.metrics.enabled=true`. |
|
||||
|
||||
Helm chart sets default probes (`initialDelaySeconds: 10`, `periodSeconds: 15`). Adjust via `console.livenessProbe` and `console.readinessProbe`.
|
||||
|
||||
---
|
||||
|
||||
## 3. Docker Compose deployment
|
||||
|
||||
Located in `deploy/compose/docker-compose.console.yaml`. Quick start:
|
||||
|
||||
```bash
|
||||
cd deploy/compose
|
||||
docker compose -f docker-compose.console.yaml --env-file console.env up -d
|
||||
```
|
||||
|
||||
`console.env` should define:
|
||||
|
||||
```
|
||||
CONSOLE_PUBLIC_BASE_URL=https://console.acme.internal
|
||||
AUTHORITY_ISSUER=https://authority.acme.internal
|
||||
AUTHORITY_CLIENT_ID=console-ui
|
||||
AUTHORITY_CLIENT_SECRET=<if using confidential client>
|
||||
AUTHORITY_SCOPES=ui.read ui.admin
|
||||
CONSOLE_GATEWAY_BASE_URL=https://api.acme.internal
|
||||
```
|
||||
|
||||
The compose bundle includes Traefik as reverse proxy with TLS termination. Update `traefik/dynamic/console.yml` for custom certificates or additional middlewares (CSP headers, rate limits).
|
||||
|
||||
---
|
||||
|
||||
## 4. Environment variables
|
||||
|
||||
| Variable | Description | Default |
|
||||
|----------|-------------|---------|
|
||||
| `CONSOLE_PUBLIC_BASE_URL` | External URL used for redirects, deep links, and telemetry. | None (required). |
|
||||
| `CONSOLE_GATEWAY_BASE_URL` | URL of the web gateway that proxies API calls (`/console/*`). | Chart service name. |
|
||||
| `AUTHORITY_ISSUER` | Authority issuer (`https://authority.example.com`). | None (required). |
|
||||
| `AUTHORITY_CLIENT_ID` | OIDC client configured in Authority. | None (required). |
|
||||
| `AUTHORITY_SCOPES` | Space-separated scopes assigned to the console client. | `ui.read ui.admin`. |
|
||||
| `AUTHORITY_DPOP_ENABLED` | Enables DPoP challenge/response (recommended true). | `true`. |
|
||||
| `CONSOLE_FEATURE_FLAGS` | Comma-separated feature flags (`runs`, `downloads.offline`, etc.). | `runs,downloads,policies`. |
|
||||
| `CONSOLE_LOG_LEVEL` | Minimum log level (`Information`, `Debug`, etc.). | `Information`. |
|
||||
| `CONSOLE_METRICS_ENABLED` | Expose `/metrics` endpoint. | `true`. |
|
||||
| `CONSOLE_SENTRY_DSN` | Optional error reporting DSN. | Blank. |
|
||||
|
||||
When running behind additional proxies, set `ASPNETCORE_FORWARDEDHEADERS_ENABLED=true` to honour `X-Forwarded-*` headers.
|
||||
|
||||
---
|
||||
|
||||
## 5. Security headers and CSP
|
||||
|
||||
The console serves a strict Content Security Policy (CSP) by default:
|
||||
|
||||
```
|
||||
default-src 'self';
|
||||
connect-src 'self' https://*.stella-ops.local;
|
||||
script-src 'self';
|
||||
style-src 'self' 'unsafe-inline';
|
||||
img-src 'self' data:;
|
||||
font-src 'self';
|
||||
frame-ancestors 'none';
|
||||
```
|
||||
|
||||
Adjust via `console.config.cspOverrides` if additional domains are required. For integrations embedding the console, update OIDC redirect URIs and Authority scopes accordingly.
|
||||
|
||||
TLS recommendations:
|
||||
|
||||
- Use TLS 1.2+ with modern cipher suite policy.
|
||||
- Enable HSTS (`Strict-Transport-Security: max-age=31536000; includeSubDomains`).
|
||||
- Provide custom trust bundles via `console.config.trustBundleSecret` when using private CAs.
|
||||
|
||||
---
|
||||
|
||||
## 6. Logging and metrics
|
||||
|
||||
- Structured logs emitted to stdout with correlation IDs. Configure log shipping via Fluent Bit or similar.
|
||||
- Metrics available at `/metrics` in Prometheus format. Key metrics include `ui_request_duration_seconds`, `ui_tenant_switch_total`, and `ui_download_manifest_refresh_seconds`.
|
||||
- Enable OpenTelemetry exporter by setting `OTEL_EXPORTER_OTLP_ENDPOINT` and associated headers in environment variables.
|
||||
|
||||
---
|
||||
|
||||
## 7. Offline and air-gap deployment
|
||||
|
||||
- Mirror container images using the Downloads workspace or Offline Kit manifest. Example:
|
||||
|
||||
```bash
|
||||
oras copy registry.stella-ops.org/stellaops/web-ui@sha256:<digest> \
|
||||
registry.airgap.local/stellaops/web-ui:2025.10.0
|
||||
```
|
||||
|
||||
- Import Offline Kit using `stella ouk import` before starting the console so manifest parity checks succeed.
|
||||
- Use `values-airgap.yaml` to disable external telemetry endpoints and configure internal certificate chains.
|
||||
- Run `helm upgrade --install` using the mirrored chart (`stellaops-<ver>.tgz`) and set `console.offlineMode=true` to surface offline banners.
|
||||
|
||||
---
|
||||
|
||||
## 8. Health checks and remediation
|
||||
|
||||
| Check | Command | Expected result |
|
||||
|-------|---------|-----------------|
|
||||
| Pod status | `kubectl get pods -n stellaops` | `Running` state with restarts = 0. |
|
||||
| Liveness | `kubectl exec deploy/stellaops-web-ui -- curl -fsS http://localhost:8080/health/live` | Returns `{"status":"Healthy"}`. |
|
||||
| Readiness | `kubectl exec deploy/stellaops-web-ui -- curl -fsS http://localhost:8080/health/ready` | Returns `{"status":"Ready"}`. |
|
||||
| Gateway reachability | `curl -I https://console.example.com/api/console/status` | `200 OK` with CSP headers. |
|
||||
| Static assets | `curl -I https://console.example.com/static/assets/app.js` | `200 OK` with long cache headers. |
|
||||
|
||||
Troubleshooting steps:
|
||||
|
||||
- **Authority unreachable:** readiness fails with `AUTHORITY_UNREACHABLE`. Check DNS, trust bundles, and Authority service health.
|
||||
- **Manifest mismatch:** console logs `DOWNLOAD_MANIFEST_SIGNATURE_INVALID`. Verify cosign key and re-sync manifest.
|
||||
- **Ingress 404:** ensure ingress controller routes host to `stellaops-web-ui` service; check TLS secret name.
|
||||
- **SSE blocked:** confirm proxy allows HTTP/1.1 and disables buffering on `/console/runs/*`.
|
||||
|
||||
---
|
||||
|
||||
## 9. References
|
||||
|
||||
- `deploy/helm/stellaops/values-*.yaml` - environment-specific overrides.
|
||||
- `deploy/compose/docker-compose.console.yaml` - Compose bundle.
|
||||
- `/docs/ui/downloads.md` - manifest and offline bundle guidance.
|
||||
- `/docs/security/console-security.md` - CSP and Authority scopes.
|
||||
- `/docs/24_OFFLINE_KIT.md` - Offline kit packaging and verification.
|
||||
- `/docs/ops/deployment-runbook.md` (pending) - wider platform deployment steps.
|
||||
|
||||
---
|
||||
|
||||
## 10. Compliance checklist
|
||||
|
||||
- [ ] Helm and Compose instructions verified against `deploy/` assets.
|
||||
- [ ] Ingress/TLS guidance aligns with Security Guild recommendations.
|
||||
- [ ] Environment variables documented with defaults and required values.
|
||||
- [ ] Health/liveness/readiness endpoints tested and listed.
|
||||
- [ ] Offline workflow (mirrors, manifest parity) captured.
|
||||
- [ ] Logging and metrics surface documented metrics.
|
||||
- [ ] CSP and security header defaults stated alongside override guidance.
|
||||
- [ ] Troubleshooting section linked to relevant runbooks.
|
||||
|
||||
---
|
||||
|
||||
*Last updated: 2025-10-27 (Sprint 23).*
|
||||
|
||||
@@ -1,160 +1,160 @@
|
||||
# Container Deployment Guide — AOC Update
|
||||
|
||||
> **Audience:** DevOps Guild, platform operators deploying StellaOps services.
|
||||
> **Scope:** Deployment configuration changes required by the Aggregation-Only Contract (AOC), including schema validators, guard environment flags, and verifier identities.
|
||||
|
||||
This guide supplements existing deployment manuals with AOC-specific configuration. It assumes familiarity with the base Compose/Helm manifests described in `ops/deployment/` and `docs/ARCHITECTURE_DEVOPS.md`.
|
||||
|
||||
---
|
||||
|
||||
## 1 · Schema validator enablement
|
||||
|
||||
### 1.1 MongoDB validators
|
||||
|
||||
- Apply JSON schema validators to `advisory_raw` and `vex_raw` collections before enabling AOC guards.
|
||||
- Before enabling validators or the idempotency index, run the duplicate audit helper to confirm no conflicting raw advisories remain:
|
||||
```bash
|
||||
mongo concelier ops/devops/scripts/check-advisory-raw-duplicates.js --eval 'var LIMIT=200;'
|
||||
```
|
||||
Resolve any reported rows prior to rollout.
|
||||
- Use the migration script provided in `ops/devops/scripts/apply-aoc-validators.js`:
|
||||
|
||||
```bash
|
||||
kubectl exec -n concelier deploy/concelier-mongo -- \
|
||||
mongo concelier ops/devops/scripts/apply-aoc-validators.js
|
||||
|
||||
kubectl exec -n excititor deploy/excititor-mongo -- \
|
||||
mongo excititor ops/devops/scripts/apply-aoc-validators.js
|
||||
```
|
||||
|
||||
- Validators enforce required fields (`tenant`, `source`, `upstream`, `linkset`) and reject forbidden keys at DB level.
|
||||
- Rollback plan: validators are applied with `validationLevel: moderate`—downgrade via the same script with `--remove` if required.
|
||||
|
||||
### 1.2 Migration order
|
||||
|
||||
1. Deploy validators in maintenance window.
|
||||
2. Roll out Concelier/Excititor images with guard middleware enabled (`AOC_GUARD_ENABLED=true`).
|
||||
3. Run smoke tests (`stella sources ingest --dry-run` fixtures) before resuming production ingestion.
|
||||
|
||||
### 1.3 Supersedes backfill verification
|
||||
|
||||
1. **Duplicate audit:** Confirm `mongo concelier ops/devops/scripts/check-advisory-raw-duplicates.js --eval 'var LIMIT=200;'` reports no conflicts before restarting Concelier with the new migrations.
|
||||
2. **Post-migration check:** After the service restarts, validate that `db.advisory` is a view pointing to `advisory_backup_20251028`:
|
||||
```bash
|
||||
mongo concelier --quiet --eval 'db.getCollectionInfos({ name: "advisory" })[0]'
|
||||
```
|
||||
The `type` should be `"view"` and `options.viewOn` should equal `"advisory_backup_20251028"`.
|
||||
3. **Supersedes chain spot-check:** Inspect a sample set to ensure deterministic chaining:
|
||||
```bash
|
||||
mongo concelier --quiet --eval '
|
||||
db.advisory_raw.aggregate([
|
||||
{ $match: { "upstream.upstream_id": { $exists: true } } },
|
||||
{ $sort: { "tenant": 1, "source.vendor": 1, "upstream.upstream_id": 1, "upstream.retrieved_at": 1 } },
|
||||
{ $limit: 5 },
|
||||
{ $project: { _id: 1, supersedes: 1 } }
|
||||
]).forEach(printjson)'
|
||||
```
|
||||
Each revision should reference the previous `_id` (or `null` for the first revision). Record findings in the change ticket before proceeding to production.
|
||||
|
||||
---
|
||||
|
||||
## 2 · Container environment flags
|
||||
|
||||
Add the following environment variables to Concelier/Excititor deployments:
|
||||
|
||||
| Variable | Default | Description |
|
||||
|----------|---------|-------------|
|
||||
| `AOC_GUARD_ENABLED` | `true` | Enables `AOCWriteGuard` interception. Set `false` only for controlled rollback. |
|
||||
| `AOC_ALLOW_SUPERSEDES_RETROFIT` | `false` | Allows temporary supersedes backfill during migration. Remove after cutover. |
|
||||
| `AOC_METRICS_ENABLED` | `true` | Emits `ingestion_write_total`, `aoc_violation_total`, etc. |
|
||||
| `AOC_TENANT_HEADER` | `X-Stella-Tenant` | Header name expected from Gateway. |
|
||||
| `AOC_VERIFIER_USER` | `stella-aoc-verify` | Read-only service user used by UI/CLI verification. |
|
||||
|
||||
Compose snippet:
|
||||
|
||||
```yaml
|
||||
environment:
|
||||
- AOC_GUARD_ENABLED=true
|
||||
- AOC_ALLOW_SUPERSEDES_RETROFIT=false
|
||||
- AOC_METRICS_ENABLED=true
|
||||
- AOC_TENANT_HEADER=X-Stella-Tenant
|
||||
- AOC_VERIFIER_USER=stella-aoc-verify
|
||||
```
|
||||
|
||||
Ensure `AOC_VERIFIER_USER` exists in Authority with `aoc:verify` scope and no write permissions.
|
||||
|
||||
---
|
||||
|
||||
## 3 · Verifier identity
|
||||
|
||||
- Create a dedicated client (`stella-aoc-verify`) via Authority bootstrap:
|
||||
|
||||
```yaml
|
||||
clients:
|
||||
- clientId: stella-aoc-verify
|
||||
grantTypes: [client_credentials]
|
||||
scopes: [aoc:verify, advisory:read, vex:read]
|
||||
tenants: [default]
|
||||
```
|
||||
|
||||
- Store credentials in secret store (`Kubernetes Secret`, `Docker swarm secret`).
|
||||
- Bind credentials to `stella aoc verify` CI jobs and Console verification service.
|
||||
- Rotate quarterly; document in `ops/authority-key-rotation.md`.
|
||||
|
||||
---
|
||||
|
||||
## 4 · Deployment steps
|
||||
|
||||
1. **Pre-checks:** Confirm database backups, alerting in maintenance mode, and staging environment validated.
|
||||
2. **Apply validators:** Run scripts per § 1.1.
|
||||
3. **Update manifests:** Inject environment variables (§ 2) and mount guard configuration configmaps.
|
||||
4. **Redeploy services:** Rolling restart Concelier/Excititor pods. Monitor `ingestion_write_total` for steady throughput.
|
||||
5. **Seed verifier:** Deploy read-only verifier user and store credentials.
|
||||
6. **Run verification:** Execute `stella aoc verify --since 24h` and ensure exit code `0`.
|
||||
7. **Update dashboards:** Point Grafana panels to new metrics (`aoc_violation_total`).
|
||||
8. **Record handoff:** Capture console screenshots and verification logs for release notes.
|
||||
|
||||
---
|
||||
|
||||
## 5 · Offline Kit updates
|
||||
|
||||
- Ship validator scripts with Offline Kit (`offline-kit/scripts/apply-aoc-validators.js`).
|
||||
- Include pre-generated verification reports for air-gapped deployments.
|
||||
- Document offline CLI workflow in bundle README referencing `docs/cli/cli-reference.md`.
|
||||
- Ensure `stella-aoc-verify` credentials are scoped to offline tenant and rotated during bundle refresh.
|
||||
|
||||
---
|
||||
|
||||
## 6 · Rollback plan
|
||||
|
||||
1. Disable guard via `AOC_GUARD_ENABLED=false` on Concelier/Excititor and rollout.
|
||||
2. Remove validators with the migration script (`--remove`).
|
||||
3. Pause verification jobs to prevent noise.
|
||||
4. Investigate and remediate upstream issues before re-enabling guards.
|
||||
|
||||
---
|
||||
|
||||
## 7 · References
|
||||
|
||||
- [Aggregation-Only Contract reference](../ingestion/aggregation-only-contract.md)
|
||||
- [Authority scopes & tenancy](../security/authority-scopes.md)
|
||||
- [Observability guide](../observability/observability.md)
|
||||
- [CLI AOC commands](../cli/cli-reference.md)
|
||||
- [Concelier architecture](../ARCHITECTURE_CONCELIER.md)
|
||||
- [Excititor architecture](../ARCHITECTURE_EXCITITOR.md)
|
||||
|
||||
---
|
||||
|
||||
## 8 · Compliance checklist
|
||||
|
||||
- [ ] Validators documented and scripts referenced for online/offline deployments.
|
||||
- [ ] Environment variables cover guard enablement, metrics, and tenant header.
|
||||
- [ ] Read-only verifier user installation steps included.
|
||||
- [ ] Offline kit instructions align with validator/verification workflow.
|
||||
- [ ] Rollback procedure captured.
|
||||
- [ ] Cross-links to AOC docs, Authority scopes, and observability guides present.
|
||||
- [ ] DevOps Guild sign-off tracked (owner: @devops-guild, due 2025-10-29).
|
||||
|
||||
---
|
||||
|
||||
*Last updated: 2025-10-26 (Sprint 19).*
|
||||
# Container Deployment Guide — AOC Update
|
||||
|
||||
> **Audience:** DevOps Guild, platform operators deploying StellaOps services.
|
||||
> **Scope:** Deployment configuration changes required by the Aggregation-Only Contract (AOC), including schema validators, guard environment flags, and verifier identities.
|
||||
|
||||
This guide supplements existing deployment manuals with AOC-specific configuration. It assumes familiarity with the base Compose/Helm manifests described in `ops/deployment/` and `docs/ARCHITECTURE_DEVOPS.md`.
|
||||
|
||||
---
|
||||
|
||||
## 1 · Schema validator enablement
|
||||
|
||||
### 1.1 MongoDB validators
|
||||
|
||||
- Apply JSON schema validators to `advisory_raw` and `vex_raw` collections before enabling AOC guards.
|
||||
- Before enabling validators or the idempotency index, run the duplicate audit helper to confirm no conflicting raw advisories remain:
|
||||
```bash
|
||||
mongo concelier ops/devops/scripts/check-advisory-raw-duplicates.js --eval 'var LIMIT=200;'
|
||||
```
|
||||
Resolve any reported rows prior to rollout.
|
||||
- Use the migration script provided in `ops/devops/scripts/apply-aoc-validators.js`:
|
||||
|
||||
```bash
|
||||
kubectl exec -n concelier deploy/concelier-mongo -- \
|
||||
mongo concelier ops/devops/scripts/apply-aoc-validators.js
|
||||
|
||||
kubectl exec -n excititor deploy/excititor-mongo -- \
|
||||
mongo excititor ops/devops/scripts/apply-aoc-validators.js
|
||||
```
|
||||
|
||||
- Validators enforce required fields (`tenant`, `source`, `upstream`, `linkset`) and reject forbidden keys at DB level.
|
||||
- Rollback plan: validators are applied with `validationLevel: moderate`—downgrade via the same script with `--remove` if required.
|
||||
|
||||
### 1.2 Migration order
|
||||
|
||||
1. Deploy validators in maintenance window.
|
||||
2. Roll out Concelier/Excititor images with guard middleware enabled (`AOC_GUARD_ENABLED=true`).
|
||||
3. Run smoke tests (`stella sources ingest --dry-run` fixtures) before resuming production ingestion.
|
||||
|
||||
### 1.3 Supersedes backfill verification
|
||||
|
||||
1. **Duplicate audit:** Confirm `mongo concelier ops/devops/scripts/check-advisory-raw-duplicates.js --eval 'var LIMIT=200;'` reports no conflicts before restarting Concelier with the new migrations.
|
||||
2. **Post-migration check:** After the service restarts, validate that `db.advisory` is a view pointing to `advisory_backup_20251028`:
|
||||
```bash
|
||||
mongo concelier --quiet --eval 'db.getCollectionInfos({ name: "advisory" })[0]'
|
||||
```
|
||||
The `type` should be `"view"` and `options.viewOn` should equal `"advisory_backup_20251028"`.
|
||||
3. **Supersedes chain spot-check:** Inspect a sample set to ensure deterministic chaining:
|
||||
```bash
|
||||
mongo concelier --quiet --eval '
|
||||
db.advisory_raw.aggregate([
|
||||
{ $match: { "upstream.upstream_id": { $exists: true } } },
|
||||
{ $sort: { "tenant": 1, "source.vendor": 1, "upstream.upstream_id": 1, "upstream.retrieved_at": 1 } },
|
||||
{ $limit: 5 },
|
||||
{ $project: { _id: 1, supersedes: 1 } }
|
||||
]).forEach(printjson)'
|
||||
```
|
||||
Each revision should reference the previous `_id` (or `null` for the first revision). Record findings in the change ticket before proceeding to production.
|
||||
|
||||
---
|
||||
|
||||
## 2 · Container environment flags
|
||||
|
||||
Add the following environment variables to Concelier/Excititor deployments:
|
||||
|
||||
| Variable | Default | Description |
|
||||
|----------|---------|-------------|
|
||||
| `AOC_GUARD_ENABLED` | `true` | Enables `AOCWriteGuard` interception. Set `false` only for controlled rollback. |
|
||||
| `AOC_ALLOW_SUPERSEDES_RETROFIT` | `false` | Allows temporary supersedes backfill during migration. Remove after cutover. |
|
||||
| `AOC_METRICS_ENABLED` | `true` | Emits `ingestion_write_total`, `aoc_violation_total`, etc. |
|
||||
| `AOC_TENANT_HEADER` | `X-Stella-Tenant` | Header name expected from Gateway. |
|
||||
| `AOC_VERIFIER_USER` | `stella-aoc-verify` | Read-only service user used by UI/CLI verification. |
|
||||
|
||||
Compose snippet:
|
||||
|
||||
```yaml
|
||||
environment:
|
||||
- AOC_GUARD_ENABLED=true
|
||||
- AOC_ALLOW_SUPERSEDES_RETROFIT=false
|
||||
- AOC_METRICS_ENABLED=true
|
||||
- AOC_TENANT_HEADER=X-Stella-Tenant
|
||||
- AOC_VERIFIER_USER=stella-aoc-verify
|
||||
```
|
||||
|
||||
Ensure `AOC_VERIFIER_USER` exists in Authority with `aoc:verify` scope and no write permissions.
|
||||
|
||||
---
|
||||
|
||||
## 3 · Verifier identity
|
||||
|
||||
- Create a dedicated client (`stella-aoc-verify`) via Authority bootstrap:
|
||||
|
||||
```yaml
|
||||
clients:
|
||||
- clientId: stella-aoc-verify
|
||||
grantTypes: [client_credentials]
|
||||
scopes: [aoc:verify, advisory:read, vex:read]
|
||||
tenants: [default]
|
||||
```
|
||||
|
||||
- Store credentials in secret store (`Kubernetes Secret`, `Docker swarm secret`).
|
||||
- Bind credentials to `stella aoc verify` CI jobs and Console verification service.
|
||||
- Rotate quarterly; document in `ops/authority-key-rotation.md`.
|
||||
|
||||
---
|
||||
|
||||
## 4 · Deployment steps
|
||||
|
||||
1. **Pre-checks:** Confirm database backups, alerting in maintenance mode, and staging environment validated.
|
||||
2. **Apply validators:** Run scripts per § 1.1.
|
||||
3. **Update manifests:** Inject environment variables (§ 2) and mount guard configuration configmaps.
|
||||
4. **Redeploy services:** Rolling restart Concelier/Excititor pods. Monitor `ingestion_write_total` for steady throughput.
|
||||
5. **Seed verifier:** Deploy read-only verifier user and store credentials.
|
||||
6. **Run verification:** Execute `stella aoc verify --since 24h` and ensure exit code `0`.
|
||||
7. **Update dashboards:** Point Grafana panels to new metrics (`aoc_violation_total`).
|
||||
8. **Record handoff:** Capture console screenshots and verification logs for release notes.
|
||||
|
||||
---
|
||||
|
||||
## 5 · Offline Kit updates
|
||||
|
||||
- Ship validator scripts with Offline Kit (`offline-kit/scripts/apply-aoc-validators.js`).
|
||||
- Include pre-generated verification reports for air-gapped deployments.
|
||||
- Document offline CLI workflow in bundle README referencing `docs/cli/cli-reference.md`.
|
||||
- Ensure `stella-aoc-verify` credentials are scoped to offline tenant and rotated during bundle refresh.
|
||||
|
||||
---
|
||||
|
||||
## 6 · Rollback plan
|
||||
|
||||
1. Disable guard via `AOC_GUARD_ENABLED=false` on Concelier/Excititor and rollout.
|
||||
2. Remove validators with the migration script (`--remove`).
|
||||
3. Pause verification jobs to prevent noise.
|
||||
4. Investigate and remediate upstream issues before re-enabling guards.
|
||||
|
||||
---
|
||||
|
||||
## 7 · References
|
||||
|
||||
- [Aggregation-Only Contract reference](../ingestion/aggregation-only-contract.md)
|
||||
- [Authority scopes & tenancy](../security/authority-scopes.md)
|
||||
- [Observability guide](../observability/observability.md)
|
||||
- [CLI AOC commands](../cli/cli-reference.md)
|
||||
- [Concelier architecture](../ARCHITECTURE_CONCELIER.md)
|
||||
- [Excititor architecture](../ARCHITECTURE_EXCITITOR.md)
|
||||
|
||||
---
|
||||
|
||||
## 8 · Compliance checklist
|
||||
|
||||
- [ ] Validators documented and scripts referenced for online/offline deployments.
|
||||
- [ ] Environment variables cover guard enablement, metrics, and tenant header.
|
||||
- [ ] Read-only verifier user installation steps included.
|
||||
- [ ] Offline kit instructions align with validator/verification workflow.
|
||||
- [ ] Rollback procedure captured.
|
||||
- [ ] Cross-links to AOC docs, Authority scopes, and observability guides present.
|
||||
- [ ] DevOps Guild sign-off tracked (owner: @devops-guild, due 2025-10-29).
|
||||
|
||||
---
|
||||
|
||||
*Last updated: 2025-10-26 (Sprint 19).*
|
||||
|
||||
@@ -1,220 +1,220 @@
|
||||
# Excititor Connector Packaging Guide
|
||||
|
||||
> **Audience:** teams implementing new Excititor provider plug‑ins (CSAF feeds,
|
||||
> OpenVEX attestations, etc.)
|
||||
> **Prerequisites:** read `docs/ARCHITECTURE_EXCITITOR.md` and the module
|
||||
> `AGENTS.md` in `src/StellaOps.Excititor.Connectors.Abstractions/`.
|
||||
|
||||
The Excititor connector SDK gives you:
|
||||
|
||||
- `VexConnectorBase` – deterministic logging, SHA‑256 helpers, time provider.
|
||||
- `VexConnectorOptionsBinder` – strongly typed YAML/JSON configuration binding.
|
||||
- `IVexConnectorOptionsValidator<T>` – custom validation hooks (offline defaults, auth invariants).
|
||||
- `VexConnectorDescriptor` & metadata helpers for consistent telemetry.
|
||||
|
||||
This guide explains how to package a connector so the Excititor Worker/WebService
|
||||
can load it via the plugin host.
|
||||
|
||||
---
|
||||
|
||||
## 1. Project layout
|
||||
|
||||
Start from the template under
|
||||
`docs/dev/templates/excititor-connector/`. It contains:
|
||||
|
||||
```
|
||||
Excititor.MyConnector/
|
||||
├── src/
|
||||
│ ├── Excititor.MyConnector.csproj
|
||||
│ ├── MyConnectorOptions.cs
|
||||
│ ├── MyConnector.cs
|
||||
│ └── MyConnectorPlugin.cs
|
||||
└── manifest/
|
||||
└── connector.manifest.yaml
|
||||
```
|
||||
|
||||
Key points:
|
||||
|
||||
- Target `net10.0`, enable `TreatWarningsAsErrors`, reference the
|
||||
`StellaOps.Excititor.Connectors.Abstractions` project (or NuGet once published).
|
||||
- Keep project ID prefix `StellaOps.Excititor.Connectors.<Provider>` so the
|
||||
plugin loader can discover it with the default search pattern.
|
||||
|
||||
### 1.1 csproj snippet
|
||||
|
||||
```xml
|
||||
<Project Sdk="Microsoft.NET.Sdk">
|
||||
<PropertyGroup>
|
||||
<TargetFramework>net10.0</TargetFramework>
|
||||
<Nullable>enable</Nullable>
|
||||
<ImplicitUsings>enable</ImplicitUsings>
|
||||
<TreatWarningsAsErrors>true</TreatWarningsAsErrors>
|
||||
</PropertyGroup>
|
||||
<ItemGroup>
|
||||
<ProjectReference Include="..\..\..\src\StellaOps.Excititor.Connectors.Abstractions\StellaOps.Excititor.Connectors.Abstractions.csproj" />
|
||||
</ItemGroup>
|
||||
</Project>
|
||||
```
|
||||
|
||||
Adjust the `ProjectReference` for your checkout (or switch to a NuGet package
|
||||
once published).
|
||||
|
||||
---
|
||||
|
||||
## 2. Implement the connector
|
||||
|
||||
1. **Options model** – create an options POCO with data-annotation attributes.
|
||||
Bind it via `VexConnectorOptionsBinder.Bind<TOptions>` in your connector
|
||||
constructor or `ValidateAsync`.
|
||||
2. **Validator** – implement `IVexConnectorOptionsValidator<TOptions>` to add
|
||||
complex checks (e.g., ensure both `clientId` and `clientSecret` are present).
|
||||
3. **Connector** – inherit from `VexConnectorBase`. Implement:
|
||||
- `ValidateAsync` – run binder/validators, log configuration summary.
|
||||
- `FetchAsync` – stream raw documents to `context.RawSink`.
|
||||
- `NormalizeAsync` – convert raw documents into `VexClaimBatch` via
|
||||
format-specific normalizers (`context.Normalizers`).
|
||||
4. **Plugin adapter** – expose the connector via a plugin entry point so the
|
||||
host can instantiate it.
|
||||
|
||||
### 2.1 Options binding example
|
||||
|
||||
```csharp
|
||||
public sealed class MyConnectorOptions
|
||||
{
|
||||
[Required]
|
||||
[Url]
|
||||
public string CatalogUri { get; set; } = default!;
|
||||
|
||||
[Required]
|
||||
public string ApiKey { get; set; } = default!;
|
||||
|
||||
[Range(1, 64)]
|
||||
public int MaxParallelRequests { get; set; } = 4;
|
||||
}
|
||||
|
||||
public sealed class MyConnectorOptionsValidator : IVexConnectorOptionsValidator<MyConnectorOptions>
|
||||
{
|
||||
public void Validate(VexConnectorDescriptor descriptor, MyConnectorOptions options, IList<string> errors)
|
||||
{
|
||||
if (!options.CatalogUri.StartsWith("https://", StringComparison.OrdinalIgnoreCase))
|
||||
{
|
||||
errors.Add("CatalogUri must use HTTPS.");
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Bind inside the connector:
|
||||
|
||||
```csharp
|
||||
private readonly MyConnectorOptions _options;
|
||||
|
||||
public MyConnector(VexConnectorDescriptor descriptor, ILogger<MyConnector> logger, TimeProvider timeProvider)
|
||||
: base(descriptor, logger, timeProvider)
|
||||
{
|
||||
// `settings` comes from the orchestrator; validators registered via DI.
|
||||
_options = VexConnectorOptionsBinder.Bind<MyConnectorOptions>(
|
||||
descriptor,
|
||||
VexConnectorSettings.Empty,
|
||||
validators: new[] { new MyConnectorOptionsValidator() });
|
||||
}
|
||||
```
|
||||
|
||||
Replace `VexConnectorSettings.Empty` with the actual settings from context
|
||||
inside `ValidateAsync`.
|
||||
|
||||
---
|
||||
|
||||
## 3. Plugin adapter & manifest
|
||||
|
||||
Create a simple plugin class that implements
|
||||
`StellaOps.Plugin.IConnectorPlugin`. The Worker/WebService plugin host uses
|
||||
this contract today.
|
||||
|
||||
```csharp
|
||||
public sealed class MyConnectorPlugin : IConnectorPlugin
|
||||
{
|
||||
private static readonly VexConnectorDescriptor Descriptor =
|
||||
new("excititor:my-provider", VexProviderKind.Vendor, "My Provider VEX");
|
||||
|
||||
public string Name => Descriptor.DisplayName;
|
||||
|
||||
public bool IsAvailable(IServiceProvider services) => true; // inject feature flags if needed
|
||||
|
||||
public IFeedConnector Create(IServiceProvider services)
|
||||
{
|
||||
var logger = services.GetRequiredService<ILogger<MyConnector>>();
|
||||
var timeProvider = services.GetRequiredService<TimeProvider>();
|
||||
return new MyConnector(Descriptor, logger, timeProvider);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
> **Note:** the Excititor Worker currently instantiates connectors through the
|
||||
> shared `IConnectorPlugin` contract. Once a dedicated Excititor plugin interface
|
||||
> lands you simply swap the base interface; the descriptor/connector code
|
||||
> remains unchanged.
|
||||
|
||||
Provide a manifest describing the assembly for operational tooling:
|
||||
|
||||
```yaml
|
||||
# manifest/connector.manifest.yaml
|
||||
id: excititor-my-provider
|
||||
assembly: StellaOps.Excititor.Connectors.MyProvider.dll
|
||||
entryPoint: StellaOps.Excititor.Connectors.MyProvider.MyConnectorPlugin
|
||||
description: >
|
||||
Official VEX feed for ExampleCorp products (CSAF JSON, daily updates).
|
||||
tags:
|
||||
- excititor
|
||||
- csaf
|
||||
- vendor
|
||||
```
|
||||
|
||||
Store manifests under `/opt/stella/excititor/plugins/<connector>/manifest/` in
|
||||
production so the deployment tooling can inventory and verify plug‑ins.
|
||||
|
||||
---
|
||||
|
||||
## 4. Packaging workflow
|
||||
|
||||
1. `dotnet publish -c Release` → copy the published DLLs to
|
||||
`/opt/stella/excititor/plugins/<Provider>/`.
|
||||
2. Place `connector.manifest.yaml` next to the binaries.
|
||||
3. Restart the Excititor Worker or WebService (hot reload not supported yet).
|
||||
4. Verify logs: `VEX-ConnectorLoader` should list the connector descriptor.
|
||||
|
||||
### 4.1 Offline kits
|
||||
|
||||
- Add the connector folder (binaries + manifest) to the Offline Kit bundle.
|
||||
- Include a `settings.sample.yaml` demonstrating offline-friendly defaults.
|
||||
- Document any external dependencies (e.g., SHA mirrors) in the manifest `notes`
|
||||
field.
|
||||
|
||||
---
|
||||
|
||||
## 5. Testing checklist
|
||||
|
||||
- Unit tests around options binding & validators.
|
||||
- Integration tests (future `StellaOps.Excititor.Connectors.Abstractions.Tests`)
|
||||
verifying deterministic logging scopes:
|
||||
`logger.BeginScope` should produce `vex.connector.id`, `vex.connector.kind`,
|
||||
and `vex.connector.operation`.
|
||||
- Deterministic SHA tests: repeated `CreateRawDocument` calls with identical
|
||||
content must return the same digest.
|
||||
|
||||
---
|
||||
|
||||
## 6. Reference template
|
||||
|
||||
See `docs/dev/templates/excititor-connector/` for the full quick‑start including:
|
||||
|
||||
- Sample options class + validator.
|
||||
- Connector implementation inheriting from `VexConnectorBase`.
|
||||
- Plugin adapter + manifest.
|
||||
|
||||
Copy the directory, rename namespaces/IDs, then iterate on provider-specific
|
||||
logic.
|
||||
|
||||
---
|
||||
|
||||
*Last updated: 2025-10-17*
|
||||
# Excititor Connector Packaging Guide
|
||||
|
||||
> **Audience:** teams implementing new Excititor provider plug‑ins (CSAF feeds,
|
||||
> OpenVEX attestations, etc.)
|
||||
> **Prerequisites:** read `docs/ARCHITECTURE_EXCITITOR.md` and the module
|
||||
> `AGENTS.md` in `src/Excititor/__Libraries/StellaOps.Excititor.Connectors.Abstractions/`.
|
||||
|
||||
The Excititor connector SDK gives you:
|
||||
|
||||
- `VexConnectorBase` – deterministic logging, SHA‑256 helpers, time provider.
|
||||
- `VexConnectorOptionsBinder` – strongly typed YAML/JSON configuration binding.
|
||||
- `IVexConnectorOptionsValidator<T>` – custom validation hooks (offline defaults, auth invariants).
|
||||
- `VexConnectorDescriptor` & metadata helpers for consistent telemetry.
|
||||
|
||||
This guide explains how to package a connector so the Excititor Worker/WebService
|
||||
can load it via the plugin host.
|
||||
|
||||
---
|
||||
|
||||
## 1. Project layout
|
||||
|
||||
Start from the template under
|
||||
`docs/dev/templates/excititor-connector/`. It contains:
|
||||
|
||||
```
|
||||
Excititor.MyConnector/
|
||||
├── src/
|
||||
│ ├── Excititor.MyConnector.csproj
|
||||
│ ├── MyConnectorOptions.cs
|
||||
│ ├── MyConnector.cs
|
||||
│ └── MyConnectorPlugin.cs
|
||||
└── manifest/
|
||||
└── connector.manifest.yaml
|
||||
```
|
||||
|
||||
Key points:
|
||||
|
||||
- Target `net10.0`, enable `TreatWarningsAsErrors`, reference the
|
||||
`StellaOps.Excititor.Connectors.Abstractions` project (or NuGet once published).
|
||||
- Keep project ID prefix `StellaOps.Excititor.Connectors.<Provider>` so the
|
||||
plugin loader can discover it with the default search pattern.
|
||||
|
||||
### 1.1 csproj snippet
|
||||
|
||||
```xml
|
||||
<Project Sdk="Microsoft.NET.Sdk">
|
||||
<PropertyGroup>
|
||||
<TargetFramework>net10.0</TargetFramework>
|
||||
<Nullable>enable</Nullable>
|
||||
<ImplicitUsings>enable</ImplicitUsings>
|
||||
<TreatWarningsAsErrors>true</TreatWarningsAsErrors>
|
||||
</PropertyGroup>
|
||||
<ItemGroup>
|
||||
<ProjectReference Include="..\..\..\src\StellaOps.Excititor.Connectors.Abstractions\StellaOps.Excititor.Connectors.Abstractions.csproj" />
|
||||
</ItemGroup>
|
||||
</Project>
|
||||
```
|
||||
|
||||
Adjust the `ProjectReference` for your checkout (or switch to a NuGet package
|
||||
once published).
|
||||
|
||||
---
|
||||
|
||||
## 2. Implement the connector
|
||||
|
||||
1. **Options model** – create an options POCO with data-annotation attributes.
|
||||
Bind it via `VexConnectorOptionsBinder.Bind<TOptions>` in your connector
|
||||
constructor or `ValidateAsync`.
|
||||
2. **Validator** – implement `IVexConnectorOptionsValidator<TOptions>` to add
|
||||
complex checks (e.g., ensure both `clientId` and `clientSecret` are present).
|
||||
3. **Connector** – inherit from `VexConnectorBase`. Implement:
|
||||
- `ValidateAsync` – run binder/validators, log configuration summary.
|
||||
- `FetchAsync` – stream raw documents to `context.RawSink`.
|
||||
- `NormalizeAsync` – convert raw documents into `VexClaimBatch` via
|
||||
format-specific normalizers (`context.Normalizers`).
|
||||
4. **Plugin adapter** – expose the connector via a plugin entry point so the
|
||||
host can instantiate it.
|
||||
|
||||
### 2.1 Options binding example
|
||||
|
||||
```csharp
|
||||
public sealed class MyConnectorOptions
|
||||
{
|
||||
[Required]
|
||||
[Url]
|
||||
public string CatalogUri { get; set; } = default!;
|
||||
|
||||
[Required]
|
||||
public string ApiKey { get; set; } = default!;
|
||||
|
||||
[Range(1, 64)]
|
||||
public int MaxParallelRequests { get; set; } = 4;
|
||||
}
|
||||
|
||||
public sealed class MyConnectorOptionsValidator : IVexConnectorOptionsValidator<MyConnectorOptions>
|
||||
{
|
||||
public void Validate(VexConnectorDescriptor descriptor, MyConnectorOptions options, IList<string> errors)
|
||||
{
|
||||
if (!options.CatalogUri.StartsWith("https://", StringComparison.OrdinalIgnoreCase))
|
||||
{
|
||||
errors.Add("CatalogUri must use HTTPS.");
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Bind inside the connector:
|
||||
|
||||
```csharp
|
||||
private readonly MyConnectorOptions _options;
|
||||
|
||||
public MyConnector(VexConnectorDescriptor descriptor, ILogger<MyConnector> logger, TimeProvider timeProvider)
|
||||
: base(descriptor, logger, timeProvider)
|
||||
{
|
||||
// `settings` comes from the orchestrator; validators registered via DI.
|
||||
_options = VexConnectorOptionsBinder.Bind<MyConnectorOptions>(
|
||||
descriptor,
|
||||
VexConnectorSettings.Empty,
|
||||
validators: new[] { new MyConnectorOptionsValidator() });
|
||||
}
|
||||
```
|
||||
|
||||
Replace `VexConnectorSettings.Empty` with the actual settings from context
|
||||
inside `ValidateAsync`.
|
||||
|
||||
---
|
||||
|
||||
## 3. Plugin adapter & manifest
|
||||
|
||||
Create a simple plugin class that implements
|
||||
`StellaOps.Plugin.IConnectorPlugin`. The Worker/WebService plugin host uses
|
||||
this contract today.
|
||||
|
||||
```csharp
|
||||
public sealed class MyConnectorPlugin : IConnectorPlugin
|
||||
{
|
||||
private static readonly VexConnectorDescriptor Descriptor =
|
||||
new("excititor:my-provider", VexProviderKind.Vendor, "My Provider VEX");
|
||||
|
||||
public string Name => Descriptor.DisplayName;
|
||||
|
||||
public bool IsAvailable(IServiceProvider services) => true; // inject feature flags if needed
|
||||
|
||||
public IFeedConnector Create(IServiceProvider services)
|
||||
{
|
||||
var logger = services.GetRequiredService<ILogger<MyConnector>>();
|
||||
var timeProvider = services.GetRequiredService<TimeProvider>();
|
||||
return new MyConnector(Descriptor, logger, timeProvider);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
> **Note:** the Excititor Worker currently instantiates connectors through the
|
||||
> shared `IConnectorPlugin` contract. Once a dedicated Excititor plugin interface
|
||||
> lands you simply swap the base interface; the descriptor/connector code
|
||||
> remains unchanged.
|
||||
|
||||
Provide a manifest describing the assembly for operational tooling:
|
||||
|
||||
```yaml
|
||||
# manifest/connector.manifest.yaml
|
||||
id: excititor-my-provider
|
||||
assembly: StellaOps.Excititor.Connectors.MyProvider.dll
|
||||
entryPoint: StellaOps.Excititor.Connectors.MyProvider.MyConnectorPlugin
|
||||
description: >
|
||||
Official VEX feed for ExampleCorp products (CSAF JSON, daily updates).
|
||||
tags:
|
||||
- excititor
|
||||
- csaf
|
||||
- vendor
|
||||
```
|
||||
|
||||
Store manifests under `/opt/stella/excititor/plugins/<connector>/manifest/` in
|
||||
production so the deployment tooling can inventory and verify plug‑ins.
|
||||
|
||||
---
|
||||
|
||||
## 4. Packaging workflow
|
||||
|
||||
1. `dotnet publish -c Release` → copy the published DLLs to
|
||||
`/opt/stella/excititor/plugins/<Provider>/`.
|
||||
2. Place `connector.manifest.yaml` next to the binaries.
|
||||
3. Restart the Excititor Worker or WebService (hot reload not supported yet).
|
||||
4. Verify logs: `VEX-ConnectorLoader` should list the connector descriptor.
|
||||
|
||||
### 4.1 Offline kits
|
||||
|
||||
- Add the connector folder (binaries + manifest) to the Offline Kit bundle.
|
||||
- Include a `settings.sample.yaml` demonstrating offline-friendly defaults.
|
||||
- Document any external dependencies (e.g., SHA mirrors) in the manifest `notes`
|
||||
field.
|
||||
|
||||
---
|
||||
|
||||
## 5. Testing checklist
|
||||
|
||||
- Unit tests around options binding & validators.
|
||||
- Integration tests (future `StellaOps.Excititor.Connectors.Abstractions.Tests`)
|
||||
verifying deterministic logging scopes:
|
||||
`logger.BeginScope` should produce `vex.connector.id`, `vex.connector.kind`,
|
||||
and `vex.connector.operation`.
|
||||
- Deterministic SHA tests: repeated `CreateRawDocument` calls with identical
|
||||
content must return the same digest.
|
||||
|
||||
---
|
||||
|
||||
## 6. Reference template
|
||||
|
||||
See `docs/dev/templates/excititor-connector/` for the full quick‑start including:
|
||||
|
||||
- Sample options class + validator.
|
||||
- Connector implementation inheriting from `VexConnectorBase`.
|
||||
- Plugin adapter + manifest.
|
||||
|
||||
Copy the directory, rename namespaces/IDs, then iterate on provider-specific
|
||||
logic.
|
||||
|
||||
---
|
||||
|
||||
*Last updated: 2025-10-17*
|
||||
|
||||
@@ -3,7 +3,7 @@
|
||||
> **Audience:** teams implementing new Vexer provider plug‑ins (CSAF feeds,
|
||||
> OpenVEX attestations, etc.)
|
||||
> **Prerequisites:** read `docs/ARCHITECTURE_VEXER.md` and the module
|
||||
> `AGENTS.md` in `src/StellaOps.Vexer.Connectors.Abstractions/`.
|
||||
> `AGENTS.md` in `src/Excititor/__Libraries/StellaOps.Excititor.Connectors.Abstractions/`.
|
||||
|
||||
The Vexer connector SDK gives you:
|
||||
|
||||
|
||||
@@ -1,212 +1,212 @@
|
||||
# Authority Plug-in Developer Guide
|
||||
|
||||
> **Status:** Updated 2025-10-11 (AUTHPLUG-DOCS-01-001) with lifecycle + limiter diagrams and refreshed rate-limit guidance aligned to PLG6 acceptance criteria.
|
||||
|
||||
## 1. Overview
|
||||
Authority plug-ins extend the **StellaOps Authority** service with custom identity providers, credential stores, and client-management logic. Unlike Concelier plug-ins (which ingest or export advisories), Authority plug-ins participate directly in authentication flows:
|
||||
|
||||
- **Use cases:** integrate corporate directories (LDAP/AD)[^ldap-rfc], delegate to external IDPs, enforce bespoke password/lockout policies, or add client provisioning automation.
|
||||
- **Constraints:** plug-ins load only during service start (no hot-reload), must function without outbound internet access, and must emit deterministic results for identical configuration input.
|
||||
- **Ship targets:** build against the host’s .NET 10 preview SDK, honour offline-first requirements, and surface actionable diagnostics so operators can triage issues from `/ready`.
|
||||
|
||||
## 2. Architecture Snapshot
|
||||
Authority hosts follow a deterministic plug-in lifecycle. The exported diagram (`docs/assets/authority/authority-plugin-lifecycle.svg`) mirrors the steps below; regenerate it from the Mermaid source if you update the flow.
|
||||
|
||||
1. **Configuration load** – `AuthorityPluginConfigurationLoader` resolves YAML manifests under `etc/authority.plugins/`.
|
||||
2. **Assembly discovery** – the shared `PluginHost` scans `StellaOps.Authority.PluginBinaries` for `StellaOps.Authority.Plugin.*.dll` assemblies.
|
||||
3. **Registrar execution** – each assembly is searched for `IAuthorityPluginRegistrar` implementations. Registrars bind options, register services, and optionally queue bootstrap tasks.
|
||||
4. **Runtime** – the host resolves `IIdentityProviderPlugin` instances, uses capability metadata to decide which OAuth grants to expose, and invokes health checks for readiness endpoints.
|
||||
|
||||
|
||||
|
||||
_Source:_ `docs/assets/authority/authority-plugin-lifecycle.mmd`
|
||||
|
||||
**Data persistence primer:** the standard Mongo-backed plugin stores users in collections named `authority_users_<pluginName>` and lockout metadata in embedded documents. Additional plugins must document their storage layout and provide deterministic collection naming to honour the Offline Kit replication process.
|
||||
|
||||
## 3. Capability Metadata
|
||||
Capability flags let the host reason about what your plug-in supports:
|
||||
|
||||
- Declare capabilities in your descriptor using the string constants from `AuthorityPluginCapabilities` (`password`, `mfa`, `clientProvisioning`, `bootstrap`). The configuration loader now validates these tokens and rejects unknown values at startup.
|
||||
- `AuthorityIdentityProviderCapabilities.FromCapabilities` projects those strings into strongly typed booleans (`SupportsPassword`, etc.). Authority Core will use these flags when wiring flows such as the password grant. Built-in plugins (e.g., Standard) will fail fast or force-enable required capabilities if the descriptor is misconfigured, so keep manifests accurate.
|
||||
- Typical configuration (`etc/authority.plugins/standard.yaml`):
|
||||
```yaml
|
||||
plugins:
|
||||
descriptors:
|
||||
standard:
|
||||
assemblyName: "StellaOps.Authority.Plugin.Standard"
|
||||
capabilities:
|
||||
- password
|
||||
- bootstrap
|
||||
```
|
||||
- Only declare a capability if the plug-in genuinely implements it. For example, if `SupportsClientProvisioning` is `true`, the plug-in must supply a working `IClientProvisioningStore`.
|
||||
|
||||
**Operational reminder:** the Authority host surfaces capability summaries during startup (see `AuthorityIdentityProviderRegistry` log lines). Use those logs during smoke tests to ensure manifests align with expectations.
|
||||
|
||||
**Configuration path normalisation:** Manifest-relative paths (e.g., `tokenSigning.keyDirectory: "../keys"`) are resolved against the YAML file location and environment variables are expanded before validation. Plug-ins should expect to receive an absolute, canonical path when options are injected.
|
||||
|
||||
**Password policy guardrails:** The Standard registrar logs a warning when a plug-in weakens the default password policy (minimum length or required character classes). Keep overrides at least as strong as the compiled defaults—operators treat the warning as an actionable security deviation.
|
||||
|
||||
## 4. Project Scaffold
|
||||
- Target **.NET 10 preview**, enable nullable, treat warnings as errors, and mark Authority plug-ins with `<IsAuthorityPlugin>true</IsAuthorityPlugin>`.
|
||||
- Minimum references:
|
||||
- `StellaOps.Authority.Plugins.Abstractions` (contracts & capability helpers)
|
||||
- `StellaOps.Plugin` (hosting/DI helpers)
|
||||
- `StellaOps.Auth.*` libraries as needed for shared token utilities (optional today).
|
||||
- Example `.csproj` (trimmed from `StellaOps.Authority.Plugin.Standard`):
|
||||
```xml
|
||||
<Project Sdk="Microsoft.NET.Sdk">
|
||||
<PropertyGroup>
|
||||
<TargetFramework>net10.0</TargetFramework>
|
||||
<Nullable>enable</Nullable>
|
||||
<TreatWarningsAsErrors>true</TreatWarningsAsErrors>
|
||||
<IsAuthorityPlugin>true</IsAuthorityPlugin>
|
||||
</PropertyGroup>
|
||||
<ItemGroup>
|
||||
<ProjectReference Include="..\StellaOps.Authority.Plugins.Abstractions\StellaOps.Authority.Plugins.Abstractions.csproj" />
|
||||
<ProjectReference Include="..\..\StellaOps.Plugin\StellaOps.Plugin.csproj" />
|
||||
</ItemGroup>
|
||||
</Project>
|
||||
```
|
||||
(Add other references—e.g., MongoDB driver, shared auth libraries—according to your implementation.)
|
||||
|
||||
## 5. Implementing `IAuthorityPluginRegistrar`
|
||||
- Create a parameterless registrar class that returns your plug-in type name via `PluginType`.
|
||||
- Use `AuthorityPluginRegistrationContext` to:
|
||||
- Bind options (`AddOptions<T>(pluginName).Bind(...)`).
|
||||
- Register singletons for stores/enrichers using manifest metadata.
|
||||
- Register any hosted bootstrap tasks (e.g., seed admin users).
|
||||
- Always validate configuration inside `PostConfigure` and throw meaningful `InvalidOperationException` to fail fast during startup.
|
||||
- Use the provided `ILoggerFactory` from DI; avoid static loggers or console writes.
|
||||
- Example skeleton:
|
||||
```csharp
|
||||
internal sealed class MyPluginRegistrar : IAuthorityPluginRegistrar
|
||||
{
|
||||
public string PluginType => "my-custom";
|
||||
|
||||
public void Register(AuthorityPluginRegistrationContext context)
|
||||
{
|
||||
var name = context.Plugin.Manifest.Name;
|
||||
|
||||
context.Services.AddOptions<MyPluginOptions>(name)
|
||||
.Bind(context.Plugin.Configuration)
|
||||
.PostConfigure(opts => opts.Validate(name));
|
||||
|
||||
context.Services.AddSingleton<IIdentityProviderPlugin>(sp =>
|
||||
new MyIdentityProvider(context.Plugin, sp.GetRequiredService<MyCredentialStore>(),
|
||||
sp.GetRequiredService<MyClaimsEnricher>(),
|
||||
sp.GetRequiredService<ILogger<MyIdentityProvider>>()));
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## 6. Identity Provider Surface
|
||||
- Implement `IIdentityProviderPlugin` to expose:
|
||||
- `IUserCredentialStore` for password validation and user CRUD.
|
||||
- `IClaimsEnricher` to append roles/attributes onto issued principals.
|
||||
- Optional `IClientProvisioningStore` for machine-to-machine clients.
|
||||
- `AuthorityIdentityProviderCapabilities` to advertise supported flows.
|
||||
- Password guidance:
|
||||
- Standard plug-in hashes via `ICryptoProvider` using Argon2id by default and emits PHC-compliant strings. Successful PBKDF2 logins trigger automatic rehashes so migrations complete gradually. See `docs/security/password-hashing.md` for tuning advice.
|
||||
- Enforce password policies before hashing to avoid storing weak credentials.
|
||||
- Health checks should probe backing stores (e.g., Mongo `ping`) and return `AuthorityPluginHealthResult` so `/ready` can surface issues.
|
||||
- When supporting additional factors (e.g., TOTP), implement `SupportsMfa` and document the enrolment flow for resource servers.
|
||||
|
||||
## 7. Configuration & Secrets
|
||||
- Authority looks for manifests under `etc/authority.plugins/`. Each YAML file maps directly to a plug-in name.
|
||||
- Support environment overrides using `STELLAOPS_AUTHORITY_PLUGINS__DESCRIPTORS__<NAME>__...`.
|
||||
- Never store raw secrets in git: allow operators to supply them via `.local.yaml`, environment variables, or injected secret files. Document which keys are mandatory.
|
||||
- Validate configuration as soon as the registrar runs; use explicit error messages to guide operators. The Standard plug-in now enforces complete bootstrap credentials (username + password) and positive lockout windows via `StandardPluginOptions.Validate`.
|
||||
- Cross-reference bootstrap workflows with `docs/ops/authority_bootstrap.md` (to be published alongside CORE6) so operators can reuse the same payload formats for manual provisioning.
|
||||
- `passwordHashing` inherits defaults from `authority.security.passwordHashing`. Override only when hardware constraints differ per plug-in:
|
||||
```yaml
|
||||
passwordHashing:
|
||||
algorithm: Argon2id
|
||||
memorySizeInKib: 19456
|
||||
iterations: 2
|
||||
parallelism: 1
|
||||
```
|
||||
Invalid values (≤0) fail fast during startup, and legacy PBKDF2 hashes rehash automatically once the new algorithm succeeds.
|
||||
|
||||
### 7.1 Token Persistence Contract
|
||||
- The host automatically persists every issued principal (access, refresh, device, authorization code) in `authority_tokens`. Plug-in code **must not** bypass this store; use the provided `IAuthorityTokenStore` helpers when implementing custom flows.
|
||||
- When a plug-in disables a subject or client outside the standard handlers, call `IAuthorityTokenStore.UpdateStatusAsync(...)` for each affected token so revocation bundles stay consistent.
|
||||
- Supply machine-friendly `revokedReason` codes (`compromised`, `rotation`, `policy`, `lifecycle`, etc.) and optional `revokedMetadata` entries when invalidating credentials. These flow straight into `revocation-bundle.json` and should remain deterministic.
|
||||
- Token scopes should be normalised (trimmed, unique, ordinal sort) before returning from plug-in verification paths. `TokenPersistenceHandlers` will keep that ordering for downstream consumers.
|
||||
|
||||
### 7.2 Claims & Enrichment Checklist
|
||||
- Authority always sets the OpenID Connect basics: `sub`, `client_id`, `preferred_username`, optional `name`, and `role` (for password flows). Plug-ins must use `IClaimsEnricher` to append additional claims in a **deterministic** order (sort arrays, normalise casing) so resource servers can rely on stable shapes.
|
||||
- Recommended enrichment keys:
|
||||
- `stellaops.realm` – plug-in/tenant identifier so services can scope policies.
|
||||
- `stellaops.subject.type` – values such as `human`, `service`, `bootstrap`.
|
||||
- `groups` / `projects` – sorted arrays describing operator entitlements.
|
||||
- Claims visible in tokens should mirror what `/token` and `/userinfo` emit. Avoid injecting sensitive PII directly; mark values with `ClassifiedString.Personal` inside the plug-in so audit sinks can tag them appropriately.
|
||||
- For client-credential flows, remember to enrich both the client principal and the validation path (`TokenValidationHandlers`) so refresh flows keep the same metadata.
|
||||
|
||||
### 7.3 Revocation Bundles & Reasons
|
||||
- Use `IAuthorityRevocationStore` to record subject/client/token revocations when credentials are deleted or rotated. Stick to the standard categories (`token`, `subject`, `client`, `key`).
|
||||
- Include a deterministic `reason` string and optional `reasonDescription` so operators understand *why* a subject was revoked when inspecting bundles offline.
|
||||
- Plug-ins should populate `metadata` with stable keys (e.g., `revokedBy`, `sourcePlugin`, `ticketId`) to simplify SOC correlation. The keys must be lowercase, ASCII, and free of secrets—bundles are mirrored to air-gapped agents.
|
||||
|
||||
## 8. Rate Limiting & Lockout Interplay
|
||||
Rate limiting and account lockouts are complementary controls. Plug-ins must surface both deterministically so operators can correlate limiter hits with credential rejections.
|
||||
|
||||
**Baseline quotas** (from `docs/dev/authority-rate-limit-tuning-outline.md`):
|
||||
|
||||
| Endpoint | Default policy | Notes |
|
||||
|----------|----------------|-------|
|
||||
| `/token` | 30 requests / 60s, queue 0 | Drop to 10/60s for untrusted ranges; raise only with WAF + monitoring. |
|
||||
| `/authorize` | 60 requests / 60s, queue 10 | Reduce carefully; interactive UX depends on headroom. |
|
||||
| `/internal/*` | Disabled by default; recommended 5/60s when enabled | Keep queue 0 for bootstrap APIs. |
|
||||
|
||||
**Retry metadata:** The middleware stamps `Retry-After` plus tags `authority.client_id`, `authority.remote_ip`, and `authority.endpoint`. Plug-ins should keep these tags intact when crafting responses or telemetry so dashboards remain consistent.
|
||||
|
||||
**Lockout counters:** Treat lockouts as **subject-scoped** decisions. When multiple instances update counters, reuse the deterministic tie-breakers documented in `src/DEDUP_CONFLICTS_RESOLUTION_ALGO.md` (freshness overrides, precedence, and stable hashes) to avoid divergent lockout states across replicas.
|
||||
|
||||
**Alerting hooks:** Emit structured logs/metrics when either the limiter or credential store rejects access. Suggested gauges include `aspnetcore_rate_limiting_rejections_total{limiter="authority-token"}` and any custom `auth.plugins.<pluginName>.lockouts_total` counter.
|
||||
|
||||
|
||||
|
||||
_Source:_ `docs/assets/authority/authority-rate-limit-flow.mmd`
|
||||
|
||||
## 9. Logging, Metrics, and Diagnostics
|
||||
- Always log via the injected `ILogger<T>`; include `pluginName` and correlation IDs where available.
|
||||
- Activity/metric names should align with `AuthorityTelemetry` constants (`service.name=stellaops-authority`).
|
||||
- Expose additional diagnostics via structured logging rather than writing custom HTTP endpoints; the host will integrate these into `/health` and `/ready`.
|
||||
- Emit metrics with stable names (`auth.plugins.<pluginName>.*`) when introducing custom instrumentation; coordinate with the Observability guild to reserve prefixes.
|
||||
|
||||
## 10. Testing & Tooling
|
||||
- Unit tests: use Mongo2Go (or similar) to exercise credential stores without hitting production infrastructure (`StandardUserCredentialStoreTests` is a template).
|
||||
- Determinism: fix timestamps to UTC and sort outputs consistently; avoid random GUIDs unless stable.
|
||||
- Smoke tests: launch `dotnet run --project src/StellaOps.Authority/StellaOps.Authority` with your plug-in under `StellaOps.Authority.PluginBinaries` and verify `/ready`.
|
||||
- Example verification snippet:
|
||||
```csharp
|
||||
[Fact]
|
||||
public async Task VerifyPasswordAsync_ReturnsSuccess()
|
||||
{
|
||||
var store = CreateCredentialStore();
|
||||
await store.UpsertUserAsync(new AuthorityUserRegistration("alice", "Pa55!", null, null, false,
|
||||
Array.Empty<string>(), new Dictionary<string, string?>()), CancellationToken.None);
|
||||
|
||||
var result = await store.VerifyPasswordAsync("alice", "Pa55!", CancellationToken.None);
|
||||
Assert.True(result.Succeeded);
|
||||
Assert.True(result.User?.Roles.Count == 0);
|
||||
}
|
||||
```
|
||||
|
||||
## 11. Packaging & Delivery
|
||||
- Output assembly should follow `StellaOps.Authority.Plugin.<Name>.dll` so the host’s search pattern picks it up.
|
||||
- Place the compiled DLL plus dependencies under `StellaOps.Authority.PluginBinaries` for offline deployments; include hashes/signatures in release notes (Security Guild guidance forthcoming).
|
||||
- Document any external prerequisites (e.g., CA cert bundle) in your plug-in README.
|
||||
- Update `etc/authority.plugins/<plugin>.yaml` samples and include deterministic SHA256 hashes for optional bootstrap payloads when distributing Offline Kit artefacts.
|
||||
|
||||
[^ldap-rfc]: Lightweight Directory Access Protocol (LDAPv3) specification — [RFC 4511](https://datatracker.ietf.org/doc/html/rfc4511).
|
||||
|
||||
## 12. Checklist & Handoff
|
||||
- ✅ Capabilities declared and validated in automated tests.
|
||||
- ✅ Bootstrap workflows documented (if `bootstrap` capability used) and repeatable.
|
||||
- ✅ Local smoke test + unit/integration suites green (`dotnet test`).
|
||||
- ✅ Operational docs updated: configuration keys, secrets guidance, troubleshooting.
|
||||
- Submit the developer guide update referencing PLG6/DOC4 and tag DevEx + Docs reviewers for sign-off.
|
||||
|
||||
---
|
||||
Mermaid sources for the embedded diagrams live under `docs/assets/authority/`. Regenerate the SVG assets with your preferred renderer before committing future updates so the visuals stay in sync with the `.mmd` definitions.
|
||||
# Authority Plug-in Developer Guide
|
||||
|
||||
> **Status:** Updated 2025-10-11 (AUTHPLUG-DOCS-01-001) with lifecycle + limiter diagrams and refreshed rate-limit guidance aligned to PLG6 acceptance criteria.
|
||||
|
||||
## 1. Overview
|
||||
Authority plug-ins extend the **StellaOps Authority** service with custom identity providers, credential stores, and client-management logic. Unlike Concelier plug-ins (which ingest or export advisories), Authority plug-ins participate directly in authentication flows:
|
||||
|
||||
- **Use cases:** integrate corporate directories (LDAP/AD)[^ldap-rfc], delegate to external IDPs, enforce bespoke password/lockout policies, or add client provisioning automation.
|
||||
- **Constraints:** plug-ins load only during service start (no hot-reload), must function without outbound internet access, and must emit deterministic results for identical configuration input.
|
||||
- **Ship targets:** build against the host’s .NET 10 preview SDK, honour offline-first requirements, and surface actionable diagnostics so operators can triage issues from `/ready`.
|
||||
|
||||
## 2. Architecture Snapshot
|
||||
Authority hosts follow a deterministic plug-in lifecycle. The exported diagram (`docs/assets/authority/authority-plugin-lifecycle.svg`) mirrors the steps below; regenerate it from the Mermaid source if you update the flow.
|
||||
|
||||
1. **Configuration load** – `AuthorityPluginConfigurationLoader` resolves YAML manifests under `etc/authority.plugins/`.
|
||||
2. **Assembly discovery** – the shared `PluginHost` scans `StellaOps.Authority.PluginBinaries` for `StellaOps.Authority.Plugin.*.dll` assemblies.
|
||||
3. **Registrar execution** – each assembly is searched for `IAuthorityPluginRegistrar` implementations. Registrars bind options, register services, and optionally queue bootstrap tasks.
|
||||
4. **Runtime** – the host resolves `IIdentityProviderPlugin` instances, uses capability metadata to decide which OAuth grants to expose, and invokes health checks for readiness endpoints.
|
||||
|
||||
|
||||
|
||||
_Source:_ `docs/assets/authority/authority-plugin-lifecycle.mmd`
|
||||
|
||||
**Data persistence primer:** the standard Mongo-backed plugin stores users in collections named `authority_users_<pluginName>` and lockout metadata in embedded documents. Additional plugins must document their storage layout and provide deterministic collection naming to honour the Offline Kit replication process.
|
||||
|
||||
## 3. Capability Metadata
|
||||
Capability flags let the host reason about what your plug-in supports:
|
||||
|
||||
- Declare capabilities in your descriptor using the string constants from `AuthorityPluginCapabilities` (`password`, `mfa`, `clientProvisioning`, `bootstrap`). The configuration loader now validates these tokens and rejects unknown values at startup.
|
||||
- `AuthorityIdentityProviderCapabilities.FromCapabilities` projects those strings into strongly typed booleans (`SupportsPassword`, etc.). Authority Core will use these flags when wiring flows such as the password grant. Built-in plugins (e.g., Standard) will fail fast or force-enable required capabilities if the descriptor is misconfigured, so keep manifests accurate.
|
||||
- Typical configuration (`etc/authority.plugins/standard.yaml`):
|
||||
```yaml
|
||||
plugins:
|
||||
descriptors:
|
||||
standard:
|
||||
assemblyName: "StellaOps.Authority.Plugin.Standard"
|
||||
capabilities:
|
||||
- password
|
||||
- bootstrap
|
||||
```
|
||||
- Only declare a capability if the plug-in genuinely implements it. For example, if `SupportsClientProvisioning` is `true`, the plug-in must supply a working `IClientProvisioningStore`.
|
||||
|
||||
**Operational reminder:** the Authority host surfaces capability summaries during startup (see `AuthorityIdentityProviderRegistry` log lines). Use those logs during smoke tests to ensure manifests align with expectations.
|
||||
|
||||
**Configuration path normalisation:** Manifest-relative paths (e.g., `tokenSigning.keyDirectory: "../keys"`) are resolved against the YAML file location and environment variables are expanded before validation. Plug-ins should expect to receive an absolute, canonical path when options are injected.
|
||||
|
||||
**Password policy guardrails:** The Standard registrar logs a warning when a plug-in weakens the default password policy (minimum length or required character classes). Keep overrides at least as strong as the compiled defaults—operators treat the warning as an actionable security deviation.
|
||||
|
||||
## 4. Project Scaffold
|
||||
- Target **.NET 10 preview**, enable nullable, treat warnings as errors, and mark Authority plug-ins with `<IsAuthorityPlugin>true</IsAuthorityPlugin>`.
|
||||
- Minimum references:
|
||||
- `StellaOps.Authority.Plugins.Abstractions` (contracts & capability helpers)
|
||||
- `StellaOps.Plugin` (hosting/DI helpers)
|
||||
- `StellaOps.Auth.*` libraries as needed for shared token utilities (optional today).
|
||||
- Example `.csproj` (trimmed from `StellaOps.Authority.Plugin.Standard`):
|
||||
```xml
|
||||
<Project Sdk="Microsoft.NET.Sdk">
|
||||
<PropertyGroup>
|
||||
<TargetFramework>net10.0</TargetFramework>
|
||||
<Nullable>enable</Nullable>
|
||||
<TreatWarningsAsErrors>true</TreatWarningsAsErrors>
|
||||
<IsAuthorityPlugin>true</IsAuthorityPlugin>
|
||||
</PropertyGroup>
|
||||
<ItemGroup>
|
||||
<ProjectReference Include="..\StellaOps.Authority.Plugins.Abstractions\StellaOps.Authority.Plugins.Abstractions.csproj" />
|
||||
<ProjectReference Include="..\..\StellaOps.Plugin\StellaOps.Plugin.csproj" />
|
||||
</ItemGroup>
|
||||
</Project>
|
||||
```
|
||||
(Add other references—e.g., MongoDB driver, shared auth libraries—according to your implementation.)
|
||||
|
||||
## 5. Implementing `IAuthorityPluginRegistrar`
|
||||
- Create a parameterless registrar class that returns your plug-in type name via `PluginType`.
|
||||
- Use `AuthorityPluginRegistrationContext` to:
|
||||
- Bind options (`AddOptions<T>(pluginName).Bind(...)`).
|
||||
- Register singletons for stores/enrichers using manifest metadata.
|
||||
- Register any hosted bootstrap tasks (e.g., seed admin users).
|
||||
- Always validate configuration inside `PostConfigure` and throw meaningful `InvalidOperationException` to fail fast during startup.
|
||||
- Use the provided `ILoggerFactory` from DI; avoid static loggers or console writes.
|
||||
- Example skeleton:
|
||||
```csharp
|
||||
internal sealed class MyPluginRegistrar : IAuthorityPluginRegistrar
|
||||
{
|
||||
public string PluginType => "my-custom";
|
||||
|
||||
public void Register(AuthorityPluginRegistrationContext context)
|
||||
{
|
||||
var name = context.Plugin.Manifest.Name;
|
||||
|
||||
context.Services.AddOptions<MyPluginOptions>(name)
|
||||
.Bind(context.Plugin.Configuration)
|
||||
.PostConfigure(opts => opts.Validate(name));
|
||||
|
||||
context.Services.AddSingleton<IIdentityProviderPlugin>(sp =>
|
||||
new MyIdentityProvider(context.Plugin, sp.GetRequiredService<MyCredentialStore>(),
|
||||
sp.GetRequiredService<MyClaimsEnricher>(),
|
||||
sp.GetRequiredService<ILogger<MyIdentityProvider>>()));
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## 6. Identity Provider Surface
|
||||
- Implement `IIdentityProviderPlugin` to expose:
|
||||
- `IUserCredentialStore` for password validation and user CRUD.
|
||||
- `IClaimsEnricher` to append roles/attributes onto issued principals.
|
||||
- Optional `IClientProvisioningStore` for machine-to-machine clients.
|
||||
- `AuthorityIdentityProviderCapabilities` to advertise supported flows.
|
||||
- Password guidance:
|
||||
- Standard plug-in hashes via `ICryptoProvider` using Argon2id by default and emits PHC-compliant strings. Successful PBKDF2 logins trigger automatic rehashes so migrations complete gradually. See `docs/security/password-hashing.md` for tuning advice.
|
||||
- Enforce password policies before hashing to avoid storing weak credentials.
|
||||
- Health checks should probe backing stores (e.g., Mongo `ping`) and return `AuthorityPluginHealthResult` so `/ready` can surface issues.
|
||||
- When supporting additional factors (e.g., TOTP), implement `SupportsMfa` and document the enrolment flow for resource servers.
|
||||
|
||||
## 7. Configuration & Secrets
|
||||
- Authority looks for manifests under `etc/authority.plugins/`. Each YAML file maps directly to a plug-in name.
|
||||
- Support environment overrides using `STELLAOPS_AUTHORITY_PLUGINS__DESCRIPTORS__<NAME>__...`.
|
||||
- Never store raw secrets in git: allow operators to supply them via `.local.yaml`, environment variables, or injected secret files. Document which keys are mandatory.
|
||||
- Validate configuration as soon as the registrar runs; use explicit error messages to guide operators. The Standard plug-in now enforces complete bootstrap credentials (username + password) and positive lockout windows via `StandardPluginOptions.Validate`.
|
||||
- Cross-reference bootstrap workflows with `docs/ops/authority_bootstrap.md` (to be published alongside CORE6) so operators can reuse the same payload formats for manual provisioning.
|
||||
- `passwordHashing` inherits defaults from `authority.security.passwordHashing`. Override only when hardware constraints differ per plug-in:
|
||||
```yaml
|
||||
passwordHashing:
|
||||
algorithm: Argon2id
|
||||
memorySizeInKib: 19456
|
||||
iterations: 2
|
||||
parallelism: 1
|
||||
```
|
||||
Invalid values (≤0) fail fast during startup, and legacy PBKDF2 hashes rehash automatically once the new algorithm succeeds.
|
||||
|
||||
### 7.1 Token Persistence Contract
|
||||
- The host automatically persists every issued principal (access, refresh, device, authorization code) in `authority_tokens`. Plug-in code **must not** bypass this store; use the provided `IAuthorityTokenStore` helpers when implementing custom flows.
|
||||
- When a plug-in disables a subject or client outside the standard handlers, call `IAuthorityTokenStore.UpdateStatusAsync(...)` for each affected token so revocation bundles stay consistent.
|
||||
- Supply machine-friendly `revokedReason` codes (`compromised`, `rotation`, `policy`, `lifecycle`, etc.) and optional `revokedMetadata` entries when invalidating credentials. These flow straight into `revocation-bundle.json` and should remain deterministic.
|
||||
- Token scopes should be normalised (trimmed, unique, ordinal sort) before returning from plug-in verification paths. `TokenPersistenceHandlers` will keep that ordering for downstream consumers.
|
||||
|
||||
### 7.2 Claims & Enrichment Checklist
|
||||
- Authority always sets the OpenID Connect basics: `sub`, `client_id`, `preferred_username`, optional `name`, and `role` (for password flows). Plug-ins must use `IClaimsEnricher` to append additional claims in a **deterministic** order (sort arrays, normalise casing) so resource servers can rely on stable shapes.
|
||||
- Recommended enrichment keys:
|
||||
- `stellaops.realm` – plug-in/tenant identifier so services can scope policies.
|
||||
- `stellaops.subject.type` – values such as `human`, `service`, `bootstrap`.
|
||||
- `groups` / `projects` – sorted arrays describing operator entitlements.
|
||||
- Claims visible in tokens should mirror what `/token` and `/userinfo` emit. Avoid injecting sensitive PII directly; mark values with `ClassifiedString.Personal` inside the plug-in so audit sinks can tag them appropriately.
|
||||
- For client-credential flows, remember to enrich both the client principal and the validation path (`TokenValidationHandlers`) so refresh flows keep the same metadata.
|
||||
|
||||
### 7.3 Revocation Bundles & Reasons
|
||||
- Use `IAuthorityRevocationStore` to record subject/client/token revocations when credentials are deleted or rotated. Stick to the standard categories (`token`, `subject`, `client`, `key`).
|
||||
- Include a deterministic `reason` string and optional `reasonDescription` so operators understand *why* a subject was revoked when inspecting bundles offline.
|
||||
- Plug-ins should populate `metadata` with stable keys (e.g., `revokedBy`, `sourcePlugin`, `ticketId`) to simplify SOC correlation. The keys must be lowercase, ASCII, and free of secrets—bundles are mirrored to air-gapped agents.
|
||||
|
||||
## 8. Rate Limiting & Lockout Interplay
|
||||
Rate limiting and account lockouts are complementary controls. Plug-ins must surface both deterministically so operators can correlate limiter hits with credential rejections.
|
||||
|
||||
**Baseline quotas** (from `docs/dev/authority-rate-limit-tuning-outline.md`):
|
||||
|
||||
| Endpoint | Default policy | Notes |
|
||||
|----------|----------------|-------|
|
||||
| `/token` | 30 requests / 60s, queue 0 | Drop to 10/60s for untrusted ranges; raise only with WAF + monitoring. |
|
||||
| `/authorize` | 60 requests / 60s, queue 10 | Reduce carefully; interactive UX depends on headroom. |
|
||||
| `/internal/*` | Disabled by default; recommended 5/60s when enabled | Keep queue 0 for bootstrap APIs. |
|
||||
|
||||
**Retry metadata:** The middleware stamps `Retry-After` plus tags `authority.client_id`, `authority.remote_ip`, and `authority.endpoint`. Plug-ins should keep these tags intact when crafting responses or telemetry so dashboards remain consistent.
|
||||
|
||||
**Lockout counters:** Treat lockouts as **subject-scoped** decisions. When multiple instances update counters, reuse the deterministic tie-breakers documented in `src/DEDUP_CONFLICTS_RESOLUTION_ALGO.md` (freshness overrides, precedence, and stable hashes) to avoid divergent lockout states across replicas.
|
||||
|
||||
**Alerting hooks:** Emit structured logs/metrics when either the limiter or credential store rejects access. Suggested gauges include `aspnetcore_rate_limiting_rejections_total{limiter="authority-token"}` and any custom `auth.plugins.<pluginName>.lockouts_total` counter.
|
||||
|
||||
|
||||
|
||||
_Source:_ `docs/assets/authority/authority-rate-limit-flow.mmd`
|
||||
|
||||
## 9. Logging, Metrics, and Diagnostics
|
||||
- Always log via the injected `ILogger<T>`; include `pluginName` and correlation IDs where available.
|
||||
- Activity/metric names should align with `AuthorityTelemetry` constants (`service.name=stellaops-authority`).
|
||||
- Expose additional diagnostics via structured logging rather than writing custom HTTP endpoints; the host will integrate these into `/health` and `/ready`.
|
||||
- Emit metrics with stable names (`auth.plugins.<pluginName>.*`) when introducing custom instrumentation; coordinate with the Observability guild to reserve prefixes.
|
||||
|
||||
## 10. Testing & Tooling
|
||||
- Unit tests: use Mongo2Go (or similar) to exercise credential stores without hitting production infrastructure (`StandardUserCredentialStoreTests` is a template).
|
||||
- Determinism: fix timestamps to UTC and sort outputs consistently; avoid random GUIDs unless stable.
|
||||
- Smoke tests: launch `dotnet run --project src/Authority/StellaOps.Authority/StellaOps.Authority` with your plug-in under `StellaOps.Authority.PluginBinaries` and verify `/ready`.
|
||||
- Example verification snippet:
|
||||
```csharp
|
||||
[Fact]
|
||||
public async Task VerifyPasswordAsync_ReturnsSuccess()
|
||||
{
|
||||
var store = CreateCredentialStore();
|
||||
await store.UpsertUserAsync(new AuthorityUserRegistration("alice", "Pa55!", null, null, false,
|
||||
Array.Empty<string>(), new Dictionary<string, string?>()), CancellationToken.None);
|
||||
|
||||
var result = await store.VerifyPasswordAsync("alice", "Pa55!", CancellationToken.None);
|
||||
Assert.True(result.Succeeded);
|
||||
Assert.True(result.User?.Roles.Count == 0);
|
||||
}
|
||||
```
|
||||
|
||||
## 11. Packaging & Delivery
|
||||
- Output assembly should follow `StellaOps.Authority.Plugin.<Name>.dll` so the host’s search pattern picks it up.
|
||||
- Place the compiled DLL plus dependencies under `StellaOps.Authority.PluginBinaries` for offline deployments; include hashes/signatures in release notes (Security Guild guidance forthcoming).
|
||||
- Document any external prerequisites (e.g., CA cert bundle) in your plug-in README.
|
||||
- Update `etc/authority.plugins/<plugin>.yaml` samples and include deterministic SHA256 hashes for optional bootstrap payloads when distributing Offline Kit artefacts.
|
||||
|
||||
[^ldap-rfc]: Lightweight Directory Access Protocol (LDAPv3) specification — [RFC 4511](https://datatracker.ietf.org/doc/html/rfc4511).
|
||||
|
||||
## 12. Checklist & Handoff
|
||||
- ✅ Capabilities declared and validated in automated tests.
|
||||
- ✅ Bootstrap workflows documented (if `bootstrap` capability used) and repeatable.
|
||||
- ✅ Local smoke test + unit/integration suites green (`dotnet test`).
|
||||
- ✅ Operational docs updated: configuration keys, secrets guidance, troubleshooting.
|
||||
- Submit the developer guide update referencing PLG6/DOC4 and tag DevEx + Docs reviewers for sign-off.
|
||||
|
||||
---
|
||||
Mermaid sources for the embedded diagrams live under `docs/assets/authority/`. Regenerate the SVG assets with your preferred renderer before committing future updates so the visuals stay in sync with the `.mmd` definitions.
|
||||
|
||||
@@ -1,115 +1,115 @@
|
||||
# BuildX Generator Quickstart
|
||||
|
||||
This quickstart explains how to run the StellaOps **BuildX SBOM generator** offline, verify the CAS handshake, and emit OCI descriptors that downstream services can attest.
|
||||
|
||||
## 1. Prerequisites
|
||||
|
||||
- Docker 25+ with BuildKit enabled (`docker buildx` available).
|
||||
- .NET 10 (preview) SDK matching the repository `global.json`.
|
||||
- Optional: network access to a StellaOps Attestor endpoint (the quickstart uses a mock service).
|
||||
|
||||
## 2. Publish the plug-in binaries
|
||||
|
||||
The BuildX generator publishes as a .NET self-contained executable with its manifest under `plugins/scanner/buildx/`.
|
||||
|
||||
```bash
|
||||
# From the repository root
|
||||
DOTNET_CLI_HOME="${PWD}/.dotnet" \
|
||||
dotnet publish src/StellaOps.Scanner.Sbomer.BuildXPlugin/StellaOps.Scanner.Sbomer.BuildXPlugin.csproj \
|
||||
-c Release \
|
||||
-o out/buildx
|
||||
```
|
||||
|
||||
# BuildX Generator Quickstart
|
||||
|
||||
This quickstart explains how to run the StellaOps **BuildX SBOM generator** offline, verify the CAS handshake, and emit OCI descriptors that downstream services can attest.
|
||||
|
||||
## 1. Prerequisites
|
||||
|
||||
- Docker 25+ with BuildKit enabled (`docker buildx` available).
|
||||
- .NET 10 (preview) SDK matching the repository `global.json`.
|
||||
- Optional: network access to a StellaOps Attestor endpoint (the quickstart uses a mock service).
|
||||
|
||||
## 2. Publish the plug-in binaries
|
||||
|
||||
The BuildX generator publishes as a .NET self-contained executable with its manifest under `plugins/scanner/buildx/`.
|
||||
|
||||
```bash
|
||||
# From the repository root
|
||||
DOTNET_CLI_HOME="${PWD}/.dotnet" \
|
||||
dotnet publish src/Scanner/StellaOps.Scanner.Sbomer.BuildXPlugin/StellaOps.Scanner.Sbomer.BuildXPlugin.csproj \
|
||||
-c Release \
|
||||
-o out/buildx
|
||||
```
|
||||
|
||||
- `out/buildx/` now contains `StellaOps.Scanner.Sbomer.BuildXPlugin.dll` and the manifest `stellaops.sbom-indexer.manifest.json`.
|
||||
- `plugins/scanner/buildx/StellaOps.Scanner.Sbomer.BuildXPlugin/` receives the same artefacts for release packaging.
|
||||
- The CI pipeline also tars and signs (SHA-256 manifest) the OS analyzer plug-ins located under
|
||||
`plugins/scanner/analyzers/os/` so they ship alongside the BuildX generator artefacts.
|
||||
|
||||
## 3. Verify the CAS handshake
|
||||
|
||||
```bash
|
||||
dotnet out/buildx/StellaOps.Scanner.Sbomer.BuildXPlugin.dll handshake \
|
||||
--manifest out/buildx \
|
||||
--cas out/cas
|
||||
```
|
||||
|
||||
The command performs a deterministic probe write (`sha256`) into the provided CAS directory and prints the resolved path.
|
||||
|
||||
## 4. Emit a descriptor + provenance placeholder
|
||||
|
||||
1. Build or identify the image you want to describe and capture its digest:
|
||||
|
||||
```bash
|
||||
docker buildx build --load -t stellaops/buildx-demo:ci samples/ci/buildx-demo
|
||||
DIGEST=$(docker image inspect stellaops/buildx-demo:ci --format '{{index .RepoDigests 0}}')
|
||||
```
|
||||
|
||||
2. Generate a CycloneDX SBOM for the built image (any tool works; here we use `docker sbom`):
|
||||
|
||||
```bash
|
||||
docker sbom stellaops/buildx-demo:ci --format cyclonedx-json > out/buildx-sbom.cdx.json
|
||||
```
|
||||
|
||||
3. Invoke the `descriptor` command, pointing at the SBOM file and optional metadata:
|
||||
|
||||
```bash
|
||||
dotnet out/buildx/StellaOps.Scanner.Sbomer.BuildXPlugin.dll descriptor \
|
||||
--manifest out/buildx \
|
||||
--image "$DIGEST" \
|
||||
--sbom out/buildx-sbom.cdx.json \
|
||||
--sbom-name buildx-sbom.cdx.json \
|
||||
--artifact-type application/vnd.stellaops.sbom.layer+json \
|
||||
--sbom-format cyclonedx-json \
|
||||
--sbom-kind inventory \
|
||||
--repository git.stella-ops.org/stellaops/buildx-demo \
|
||||
--build-ref $(git rev-parse HEAD) \
|
||||
> out/buildx-descriptor.json
|
||||
```
|
||||
|
||||
|
||||
## 3. Verify the CAS handshake
|
||||
|
||||
```bash
|
||||
dotnet out/buildx/StellaOps.Scanner.Sbomer.BuildXPlugin.dll handshake \
|
||||
--manifest out/buildx \
|
||||
--cas out/cas
|
||||
```
|
||||
|
||||
The command performs a deterministic probe write (`sha256`) into the provided CAS directory and prints the resolved path.
|
||||
|
||||
## 4. Emit a descriptor + provenance placeholder
|
||||
|
||||
1. Build or identify the image you want to describe and capture its digest:
|
||||
|
||||
```bash
|
||||
docker buildx build --load -t stellaops/buildx-demo:ci samples/ci/buildx-demo
|
||||
DIGEST=$(docker image inspect stellaops/buildx-demo:ci --format '{{index .RepoDigests 0}}')
|
||||
```
|
||||
|
||||
2. Generate a CycloneDX SBOM for the built image (any tool works; here we use `docker sbom`):
|
||||
|
||||
```bash
|
||||
docker sbom stellaops/buildx-demo:ci --format cyclonedx-json > out/buildx-sbom.cdx.json
|
||||
```
|
||||
|
||||
3. Invoke the `descriptor` command, pointing at the SBOM file and optional metadata:
|
||||
|
||||
```bash
|
||||
dotnet out/buildx/StellaOps.Scanner.Sbomer.BuildXPlugin.dll descriptor \
|
||||
--manifest out/buildx \
|
||||
--image "$DIGEST" \
|
||||
--sbom out/buildx-sbom.cdx.json \
|
||||
--sbom-name buildx-sbom.cdx.json \
|
||||
--artifact-type application/vnd.stellaops.sbom.layer+json \
|
||||
--sbom-format cyclonedx-json \
|
||||
--sbom-kind inventory \
|
||||
--repository git.stella-ops.org/stellaops/buildx-demo \
|
||||
--build-ref $(git rev-parse HEAD) \
|
||||
> out/buildx-descriptor.json
|
||||
```
|
||||
|
||||
The output JSON captures:
|
||||
|
||||
- OCI artifact descriptor including size, digest, and annotations (`org.stellaops.*`).
|
||||
- Provenance placeholder (`expectedDsseSha256`, `nonce`, `attestorUri` when provided). `nonce` is derived deterministically from the image + SBOM metadata so repeated runs produce identical placeholders for identical inputs.
|
||||
- Generator metadata and deterministic timestamps.
|
||||
|
||||
## 5. (Optional) Send the placeholder to an Attestor
|
||||
|
||||
The plug-in can POST the descriptor metadata to an Attestor endpoint, returning once it receives an HTTP 202.
|
||||
|
||||
```bash
|
||||
python3 - <<'PY' &
|
||||
from http.server import BaseHTTPRequestHandler, HTTPServer
|
||||
class Handler(BaseHTTPRequestHandler):
|
||||
def do_POST(self):
|
||||
_ = self.rfile.read(int(self.headers.get('Content-Length', 0)))
|
||||
self.send_response(202); self.end_headers(); self.wfile.write(b'accepted')
|
||||
def log_message(self, fmt, *args):
|
||||
return
|
||||
server = HTTPServer(('127.0.0.1', 8085), Handler)
|
||||
try:
|
||||
server.serve_forever()
|
||||
except KeyboardInterrupt:
|
||||
pass
|
||||
finally:
|
||||
server.server_close()
|
||||
PY
|
||||
MOCK_PID=$!
|
||||
|
||||
dotnet out/buildx/StellaOps.Scanner.Sbomer.BuildXPlugin.dll descriptor \
|
||||
--manifest out/buildx \
|
||||
--image "$DIGEST" \
|
||||
--sbom out/buildx-sbom.cdx.json \
|
||||
--attestor http://127.0.0.1:8085/provenance \
|
||||
--attestor-token "$STELLAOPS_ATTESTOR_TOKEN" \
|
||||
> out/buildx-descriptor.json
|
||||
|
||||
kill $MOCK_PID
|
||||
```
|
||||
|
||||
Set `STELLAOPS_ATTESTOR_TOKEN` (or pass `--attestor-token`) when the Attestor requires bearer authentication. Use `--attestor-insecure` for lab environments with self-signed certificates.
|
||||
|
||||
## 6. CI workflow example
|
||||
|
||||
|
||||
## 5. (Optional) Send the placeholder to an Attestor
|
||||
|
||||
The plug-in can POST the descriptor metadata to an Attestor endpoint, returning once it receives an HTTP 202.
|
||||
|
||||
```bash
|
||||
python3 - <<'PY' &
|
||||
from http.server import BaseHTTPRequestHandler, HTTPServer
|
||||
class Handler(BaseHTTPRequestHandler):
|
||||
def do_POST(self):
|
||||
_ = self.rfile.read(int(self.headers.get('Content-Length', 0)))
|
||||
self.send_response(202); self.end_headers(); self.wfile.write(b'accepted')
|
||||
def log_message(self, fmt, *args):
|
||||
return
|
||||
server = HTTPServer(('127.0.0.1', 8085), Handler)
|
||||
try:
|
||||
server.serve_forever()
|
||||
except KeyboardInterrupt:
|
||||
pass
|
||||
finally:
|
||||
server.server_close()
|
||||
PY
|
||||
MOCK_PID=$!
|
||||
|
||||
dotnet out/buildx/StellaOps.Scanner.Sbomer.BuildXPlugin.dll descriptor \
|
||||
--manifest out/buildx \
|
||||
--image "$DIGEST" \
|
||||
--sbom out/buildx-sbom.cdx.json \
|
||||
--attestor http://127.0.0.1:8085/provenance \
|
||||
--attestor-token "$STELLAOPS_ATTESTOR_TOKEN" \
|
||||
> out/buildx-descriptor.json
|
||||
|
||||
kill $MOCK_PID
|
||||
```
|
||||
|
||||
Set `STELLAOPS_ATTESTOR_TOKEN` (or pass `--attestor-token`) when the Attestor requires bearer authentication. Use `--attestor-insecure` for lab environments with self-signed certificates.
|
||||
|
||||
## 6. CI workflow example
|
||||
|
||||
A reusable GitHub Actions workflow is provided under `samples/ci/buildx-demo/github-actions-buildx-demo.yml`. It publishes the plug-in, runs the handshake, builds the demo image, emits a descriptor, and uploads both the descriptor and the mock-Attestor request as artefacts.
|
||||
|
||||
Add the workflow to your repository (or call it via `workflow_call`) and adjust the SBOM path + Attestor URL as needed. The workflow also re-runs the `descriptor` command and diffs the results (ignoring the `generatedAt` timestamp) so you catch regressions that would break deterministic CI use.
|
||||
|
||||
@@ -1,86 +1,86 @@
|
||||
# Excititor Statement Backfill Runbook
|
||||
|
||||
Last updated: 2025-10-19
|
||||
|
||||
## Overview
|
||||
|
||||
Use this runbook when you need to rebuild the `vex.statements` collection from historical raw documents. Typical scenarios:
|
||||
|
||||
- Upgrading the statement schema (e.g., adding severity/KEV/EPSS signals).
|
||||
- Recovering from a partial ingest outage where statements were never persisted.
|
||||
- Seeding a freshly provisioned Excititor deployment from an existing raw archive.
|
||||
|
||||
Backfill operates server-side via the Excititor WebService and reuses the same pipeline that powers the `/excititor/statements` ingestion endpoint. Each raw document is normalized, signed metadata is preserved, and duplicate statements are skipped unless the run is forced.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
1. **Connectivity to Excititor WebService** – the CLI uses the backend URL configured in `stellaops.yml` or the `--backend-url` argument.
|
||||
2. **Authority credentials** – the CLI honours the existing Authority client configuration; ensure the caller has permission to invoke admin endpoints.
|
||||
3. **Mongo replica set** (recommended) – causal consistency guarantees rely on majority read/write concerns. Standalone deployment works but skips cross-document transactions.
|
||||
|
||||
## CLI command
|
||||
|
||||
```
|
||||
stellaops excititor backfill-statements \
|
||||
[--retrieved-since <ISO8601>] \
|
||||
[--force] \
|
||||
[--batch-size <int>] \
|
||||
[--max-documents <int>]
|
||||
```
|
||||
|
||||
| Option | Description |
|
||||
| ------ | ----------- |
|
||||
| `--retrieved-since` | Only process raw documents fetched on or after the specified timestamp (UTC by default). |
|
||||
| `--force` | Reprocess documents even if matching statements already exist (useful after schema upgrades). |
|
||||
| `--batch-size` | Number of raw documents pulled per batch (default `100`). |
|
||||
| `--max-documents` | Optional hard limit on the number of raw documents to evaluate. |
|
||||
|
||||
Example – replay the last 48 hours of Red Hat ingest while keeping existing statements:
|
||||
|
||||
```
|
||||
stellaops excititor backfill-statements \
|
||||
--retrieved-since "$(date -u -d '48 hours ago' +%Y-%m-%dT%H:%M:%SZ)"
|
||||
```
|
||||
|
||||
Example – full replay with forced overwrites, capped at 2,000 documents:
|
||||
|
||||
```
|
||||
stellaops excititor backfill-statements --force --max-documents 2000
|
||||
```
|
||||
|
||||
The command returns a summary similar to:
|
||||
|
||||
```
|
||||
Backfill completed: evaluated 450, backfilled 180, claims written 320, skipped 270, failures 0.
|
||||
```
|
||||
|
||||
## Behaviour
|
||||
|
||||
- Raw documents are streamed in ascending `retrievedAt` order.
|
||||
- Each document is normalized using the registered VEX normalizers (CSAF, CycloneDX, OpenVEX).
|
||||
- Statements are appended through the same `IVexClaimStore.AppendAsync` path that powers `/excititor/statements`.
|
||||
- Duplicate detection compares `Document.Digest`; duplicates are skipped unless `--force` is specified.
|
||||
- Failures are logged with the offending digest and continue with the next document.
|
||||
|
||||
## Observability
|
||||
|
||||
- CLI logs aggregate counts and the backend logs per-digest warnings or errors.
|
||||
- Mongo writes carry majority write concern; expect backfill throughput to match ingest baselines (≈5 seconds warm, 30 seconds cold).
|
||||
- Monitor the `excititor.storage.backfill` log scope for detailed telemetry.
|
||||
|
||||
## Post-run verification
|
||||
|
||||
1. Inspect the `vex.statements` collection for the targeted window (check `InsertedAt`).
|
||||
2. Re-run the Excititor storage test suite if possible:
|
||||
```
|
||||
dotnet test src/StellaOps.Excititor.Storage.Mongo.Tests/StellaOps.Excititor.Storage.Mongo.Tests.csproj
|
||||
```
|
||||
3. Optionally, call `/excititor/statements/{vulnerabilityId}/{productKey}` to confirm the expected statements exist.
|
||||
|
||||
## Rollback
|
||||
|
||||
If a forced run produced incorrect statements, use the standard Mongo rollback procedure:
|
||||
|
||||
1. Identify the `InsertedAt` window for the backfill run.
|
||||
2. Delete affected records from `vex.statements` (and any downstream exports if applicable).
|
||||
3. Rerun the backfill command with corrected parameters.
|
||||
# Excititor Statement Backfill Runbook
|
||||
|
||||
Last updated: 2025-10-19
|
||||
|
||||
## Overview
|
||||
|
||||
Use this runbook when you need to rebuild the `vex.statements` collection from historical raw documents. Typical scenarios:
|
||||
|
||||
- Upgrading the statement schema (e.g., adding severity/KEV/EPSS signals).
|
||||
- Recovering from a partial ingest outage where statements were never persisted.
|
||||
- Seeding a freshly provisioned Excititor deployment from an existing raw archive.
|
||||
|
||||
Backfill operates server-side via the Excititor WebService and reuses the same pipeline that powers the `/excititor/statements` ingestion endpoint. Each raw document is normalized, signed metadata is preserved, and duplicate statements are skipped unless the run is forced.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
1. **Connectivity to Excititor WebService** – the CLI uses the backend URL configured in `stellaops.yml` or the `--backend-url` argument.
|
||||
2. **Authority credentials** – the CLI honours the existing Authority client configuration; ensure the caller has permission to invoke admin endpoints.
|
||||
3. **Mongo replica set** (recommended) – causal consistency guarantees rely on majority read/write concerns. Standalone deployment works but skips cross-document transactions.
|
||||
|
||||
## CLI command
|
||||
|
||||
```
|
||||
stellaops excititor backfill-statements \
|
||||
[--retrieved-since <ISO8601>] \
|
||||
[--force] \
|
||||
[--batch-size <int>] \
|
||||
[--max-documents <int>]
|
||||
```
|
||||
|
||||
| Option | Description |
|
||||
| ------ | ----------- |
|
||||
| `--retrieved-since` | Only process raw documents fetched on or after the specified timestamp (UTC by default). |
|
||||
| `--force` | Reprocess documents even if matching statements already exist (useful after schema upgrades). |
|
||||
| `--batch-size` | Number of raw documents pulled per batch (default `100`). |
|
||||
| `--max-documents` | Optional hard limit on the number of raw documents to evaluate. |
|
||||
|
||||
Example – replay the last 48 hours of Red Hat ingest while keeping existing statements:
|
||||
|
||||
```
|
||||
stellaops excititor backfill-statements \
|
||||
--retrieved-since "$(date -u -d '48 hours ago' +%Y-%m-%dT%H:%M:%SZ)"
|
||||
```
|
||||
|
||||
Example – full replay with forced overwrites, capped at 2,000 documents:
|
||||
|
||||
```
|
||||
stellaops excititor backfill-statements --force --max-documents 2000
|
||||
```
|
||||
|
||||
The command returns a summary similar to:
|
||||
|
||||
```
|
||||
Backfill completed: evaluated 450, backfilled 180, claims written 320, skipped 270, failures 0.
|
||||
```
|
||||
|
||||
## Behaviour
|
||||
|
||||
- Raw documents are streamed in ascending `retrievedAt` order.
|
||||
- Each document is normalized using the registered VEX normalizers (CSAF, CycloneDX, OpenVEX).
|
||||
- Statements are appended through the same `IVexClaimStore.AppendAsync` path that powers `/excititor/statements`.
|
||||
- Duplicate detection compares `Document.Digest`; duplicates are skipped unless `--force` is specified.
|
||||
- Failures are logged with the offending digest and continue with the next document.
|
||||
|
||||
## Observability
|
||||
|
||||
- CLI logs aggregate counts and the backend logs per-digest warnings or errors.
|
||||
- Mongo writes carry majority write concern; expect backfill throughput to match ingest baselines (≈5 seconds warm, 30 seconds cold).
|
||||
- Monitor the `excititor.storage.backfill` log scope for detailed telemetry.
|
||||
|
||||
## Post-run verification
|
||||
|
||||
1. Inspect the `vex.statements` collection for the targeted window (check `InsertedAt`).
|
||||
2. Re-run the Excititor storage test suite if possible:
|
||||
```
|
||||
dotnet test src/Excititor/__Tests/StellaOps.Excititor.Storage.Mongo.Tests/StellaOps.Excititor.Storage.Mongo.Tests.csproj
|
||||
```
|
||||
3. Optionally, call `/excititor/statements/{vulnerabilityId}/{productKey}` to confirm the expected statements exist.
|
||||
|
||||
## Rollback
|
||||
|
||||
If a forced run produced incorrect statements, use the standard Mongo rollback procedure:
|
||||
|
||||
1. Identify the `InsertedAt` window for the backfill run.
|
||||
2. Delete affected records from `vex.statements` (and any downstream exports if applicable).
|
||||
3. Rerun the backfill command with corrected parameters.
|
||||
|
||||
@@ -1,146 +1,146 @@
|
||||
# Authority DPoP & mTLS Implementation Plan (2025-10-19)
|
||||
|
||||
## Purpose
|
||||
- Provide the implementation blueprint for AUTH-DPOP-11-001 and AUTH-MTLS-11-002.
|
||||
- Unify sender-constraint validation across Authority, downstream services, and clients.
|
||||
- Capture deterministic, testable steps that unblock UI/Signer guilds depending on DPoP/mTLS hardening.
|
||||
|
||||
## Scope
|
||||
- Token endpoint validation, issuance, and storage changes inside `StellaOps.Authority`.
|
||||
- Shared security primitives consumed by Authority, Scanner, Signer, CLI, and UI.
|
||||
- Operator-facing configuration, auditing, and observability.
|
||||
- Out of scope: PoE enforcement (Signer) and CLI/UI client UX; those teams consume the new capabilities.
|
||||
|
||||
> **Status update (2025-10-19):** `ValidateDpopProofHandler`, `AuthorityClientCertificateValidator`, and the supporting storage/audit plumbing now live in `src/StellaOps.Authority`. DPoP proofs populate `cnf.jkt`, mTLS bindings enforce certificate thumbprints via `cnf.x5t#S256`, and token documents persist the sender constraint metadata. In-memory nonce issuance is wired (Redis implementation to follow). Documentation and configuration references were updated (`docs/11_AUTHORITY.md`). Targeted unit/integration tests were added; running the broader test suite is currently blocked by pre-existing `StellaOps.Concelier.Storage.Mongo` build errors.
|
||||
# Authority DPoP & mTLS Implementation Plan (2025-10-19)
|
||||
|
||||
## Purpose
|
||||
- Provide the implementation blueprint for AUTH-DPOP-11-001 and AUTH-MTLS-11-002.
|
||||
- Unify sender-constraint validation across Authority, downstream services, and clients.
|
||||
- Capture deterministic, testable steps that unblock UI/Signer guilds depending on DPoP/mTLS hardening.
|
||||
|
||||
## Scope
|
||||
- Token endpoint validation, issuance, and storage changes inside `StellaOps.Authority`.
|
||||
- Shared security primitives consumed by Authority, Scanner, Signer, CLI, and UI.
|
||||
- Operator-facing configuration, auditing, and observability.
|
||||
- Out of scope: PoE enforcement (Signer) and CLI/UI client UX; those teams consume the new capabilities.
|
||||
|
||||
> **Status update (2025-10-19):** `ValidateDpopProofHandler`, `AuthorityClientCertificateValidator`, and the supporting storage/audit plumbing now live in `src/Authority/StellaOps.Authority`. DPoP proofs populate `cnf.jkt`, mTLS bindings enforce certificate thumbprints via `cnf.x5t#S256`, and token documents persist the sender constraint metadata. In-memory nonce issuance is wired (Redis implementation to follow). Documentation and configuration references were updated (`docs/11_AUTHORITY.md`). Targeted unit/integration tests were added; running the broader test suite is currently blocked by pre-existing `StellaOps.Concelier.Storage.Mongo` build errors.
|
||||
>
|
||||
> **Status update (2025-10-20):** Redis-backed nonce configuration is exposed through `security.senderConstraints.dpop.nonce` with sample YAML (`etc/authority.yaml.sample`) and architecture docs refreshed. Operator guide now includes concrete Redis/required audiences snippet; nonce challenge regression remains covered by `ValidateDpopProof_IssuesNonceChallenge_WhenNonceMissing`.
|
||||
>
|
||||
> **Status update (2025-10-23):** mTLS enforcement now honours `security.senderConstraints.mtls.enforceForAudiences`, automatically rejecting non-mTLS clients targeting audiences such as `signer`. Certificate bindings validate thumbprint, issuer, subject, serial number, and SAN values, producing deterministic error codes for operators. Introspection responses include `cnf.x5t#S256`, and new unit tests cover audience enforcement, binding mismatches, and bootstrap storage. Docs/sample config updated accordingly.
|
||||
|
||||
## Design Summary
|
||||
- Extract the existing Scanner `DpopProofValidator` stack into a shared `StellaOps.Auth.Security` library used by Authority and resource servers.
|
||||
- Extend Authority configuration (`authority.yaml`) with strongly-typed `senderConstraints.dpop` and `senderConstraints.mtls` sections (map to sample already shown in architecture doc).
|
||||
- Require DPoP proofs on `/token` when the registered client policy is `senderConstraint=dpop`; bind issued access tokens via `cnf.jkt`.
|
||||
- Introduce Authority-managed nonce issuance for “high value” audiences (default: `signer`, `attestor`) with Redis-backed persistence and deterministic auditing.
|
||||
- Enable OAuth 2.0 mTLS (RFC 8705) by storing certificate bindings per client, requesting client certificates at TLS termination, and stamping `cnf.x5t#S256` into issued tokens plus introspection output.
|
||||
- Surface structured logs and counters for both DPoP and mTLS flows; provide integration tests that cover success, replay, invalid proof, and certificate mismatch cases.
|
||||
|
||||
## AUTH-DPOP-11-001 — Proof Validation & Nonce Handling
|
||||
|
||||
**Shared validator**
|
||||
- Move `DpopProofValidator`, option types, and replay cache interfaces from `StellaOps.Scanner.Core` into a new assembly `StellaOps.Auth.Security`.
|
||||
- Provide pluggable caches: `InMemoryDpopReplayCache` (existing) and new `RedisDpopReplayCache` (leveraging the Authority Redis connection).
|
||||
- Ensure the validator exposes the validated `SecurityKey`, `jti`, and `iat` so Authority can construct the `cnf` claim and compute nonce expiry.
|
||||
|
||||
**Configuration model**
|
||||
- Extend `StellaOpsAuthorityOptions.Security` with a `SenderConstraints` property containing:
|
||||
- `Dpop` (`enabled`, `allowedAlgorithms`, `maxAgeSeconds`, `clockSkewSeconds`, `replayWindowSeconds`, `nonce` settings with `enabled`, `ttlSeconds`, `requiredAudiences`, `maxIssuancePerMinute`).
|
||||
- `Mtls` (`enabled`, `requireChainValidation`, `clientCaBundle`, `allowedSubjectPatterns`, `allowedSanTypes`).
|
||||
- Bind from YAML (`authority.security.senderConstraints.*`) while preserving backwards compatibility (defaults keep both disabled).
|
||||
|
||||
**Token endpoint pipeline**
|
||||
- Introduce a scoped OpenIddict handler `ValidateDpopProofHandler` inserted before `ValidateClientCredentialsHandler`.
|
||||
- Determine the required sender constraint from client metadata:
|
||||
- Add `AuthorityClientMetadataKeys.SenderConstraint` storing `dpop` or `mtls`.
|
||||
- Optionally allow per-client overrides for nonce requirement.
|
||||
- When `dpop` is required:
|
||||
- Read the `DPoP` header from the ASP.NET request, reject with `invalid_token` + `WWW-Authenticate: DPoP error="invalid_dpop_proof"` if absent.
|
||||
- Call the shared validator with method/URI. Enforce algorithm allowlist and `iat` window from options.
|
||||
- Persist the `jkt` thumbprint plus replay cache state in the OpenIddict transaction (`AuthorityOpenIddictConstants.DpopKeyThumbprintProperty`, `DpopIssuedAtProperty`).
|
||||
- When the requested audience intersects `SenderConstraints.Dpop.Nonce.RequiredAudiences`, require `nonce` in the proof; on first failure respond with HTTP 401, `error="use_dpop_nonce"`, and include `DPoP-Nonce` header (see nonce note below). Cache the rejection reason for audit logging.
|
||||
|
||||
**Nonce service**
|
||||
- Add `IDpopNonceStore` with methods `IssueAsync(audience, clientId, jkt)` and `TryConsumeAsync(nonce, audience, clientId, jkt)`.
|
||||
- Default implementation `RedisDpopNonceStore` storing SHA-256 hashes of nonces keyed by `audience:clientId:jkt`. TTL comes from `SenderConstraints.Dpop.Nonce.Ttl`.
|
||||
- Create helper `DpopNonceIssuer` used by `ValidateDpopProofHandler` to issue nonces when missing/expired, enforcing issuance rate limits (per options) and tagging audit/log records.
|
||||
- On successful validation (nonce supplied and consumed) stamp metadata into the transaction for auditing.
|
||||
- Update `ClientCredentialsHandlers` to observe nonce enforcement: when a nonce challenge was sent, emit structured audit with `nonce_issued`, `audiences`, and `retry`.
|
||||
|
||||
**Token issuance**
|
||||
- In `HandleClientCredentialsHandler`, if the transaction contains a validated DPoP key:
|
||||
- Build `cnf.jkt` using thumbprint from validator.
|
||||
- Include `auth_time`/`dpop_jti` as needed for diagnostics.
|
||||
- Persist the thumbprint alongside token metadata in Mongo (extend `AuthorityTokenDocument` with `SenderConstraint`, `KeyThumbprint`, `Nonce` fields).
|
||||
|
||||
**Auditing & observability**
|
||||
- Emit new audit events:
|
||||
- `authority.dpop.proof.validated` (success/failure, clientId, audience, thumbprint, nonce status, jti).
|
||||
- `authority.dpop.nonce.issued` and `authority.dpop.nonce.consumed`.
|
||||
- Metrics (Prometheus style):
|
||||
- `authority_dpop_validations_total{result,reason}`.
|
||||
- `authority_dpop_nonce_issued_total{audience}` and `authority_dpop_nonce_fails_total{reason}`.
|
||||
- Structured logs include `authority.sender_constraint=dpop`, `authority.dpop_thumbprint`, `authority.dpop_nonce`.
|
||||
|
||||
**Testing**
|
||||
- Unit tests for the handler pipeline using fake OpenIddict transactions.
|
||||
- Replay/nonce tests with in-memory and Redis stores.
|
||||
- Integration tests in `StellaOps.Authority.Tests` covering:
|
||||
- Valid DPoP proof issuing `cnf.jkt`.
|
||||
- Missing header → challenge with nonce.
|
||||
- Replayed `jti` rejected.
|
||||
- Invalid nonce rejected even after issuance.
|
||||
- Contract tests to ensure `/.well-known/openid-configuration` advertises `dpop_signing_alg_values_supported` and `dpop_nonce_supported` when enabled.
|
||||
|
||||
## AUTH-MTLS-11-002 — Certificate-Bound Tokens
|
||||
|
||||
**Configuration model**
|
||||
- Reuse `SenderConstraints.Mtls` described above; include:
|
||||
- `enforceForAudiences` list (defaults `signer`, `attestor`, `scheduler`).
|
||||
- `certificateRotationGraceSeconds` for overlap.
|
||||
- `allowedClientCertificateAuthorities` absolute paths.
|
||||
|
||||
**Kestrel/TLS pipeline**
|
||||
- Configure Kestrel with `ClientCertificateMode.AllowCertificate` globally and implement middleware that enforces certificate presence only when the resolved client requires mTLS.
|
||||
- Add `IAuthorityClientCertificateValidator` that validates presented certificate chain, SANs (`dns`, `uri`, optional SPIFFE), and thumbprint matches one of the stored bindings.
|
||||
- Cache validation results per connection id to avoid rehashing on every request.
|
||||
|
||||
**Client registration & storage**
|
||||
- Extend `AuthorityClientDocument` with `List<AuthorityClientCertificateBinding>` containing:
|
||||
- `Thumbprint`, `SerialNumber`, `Subject`, `NotBefore`, `NotAfter`, `Sans`, `CreatedAt`, `UpdatedAt`, `Label`.
|
||||
- Provide admin API mutations (`/admin/clients/{id}/certificates`) for ops tooling (deferred implementation but schema ready).
|
||||
- Update plugin provisioning store (`StandardClientProvisioningStore`) to map descriptors with certificate bindings and `senderConstraint`.
|
||||
- Persist binding state in Mongo migrations (index on `{clientId, thumbprint}`).
|
||||
|
||||
**Token issuance & introspection**
|
||||
- Add a transaction property capturing the validated client certificate thumbprint.
|
||||
- `HandleClientCredentialsHandler`:
|
||||
- When mTLS required, ensure certificate info present; reject otherwise.
|
||||
- Stamp `cnf` claim: `principal.SetClaim("cnf", JsonSerializer.Serialize(new { x5t#S256 = thumbprint }))`.
|
||||
- Store binding metadata in issued token document for audit.
|
||||
- Update `ValidateAccessTokenHandler` and introspection responses to surface `cnf.x5t#S256`.
|
||||
- Ensure refresh tokens (if ever enabled) copy the binding data.
|
||||
|
||||
**Auditing & observability**
|
||||
- Audit events:
|
||||
- `authority.mtls.handshake` (success/failure, clientId, thumbprint, issuer, subject).
|
||||
- `authority.mtls.binding.missing` when a required client posts without a cert.
|
||||
- Metrics:
|
||||
- `authority_mtls_handshakes_total{result}`.
|
||||
- `authority_mtls_certificate_rotations_total`.
|
||||
- Logs include `authority.sender_constraint=mtls`, `authority.mtls_thumbprint`, `authority.mtls_subject`.
|
||||
|
||||
**Testing**
|
||||
- Unit tests for certificate validation rules (SAN mismatches, expiry, CA trust).
|
||||
- Integration tests running Kestrel with test certificates:
|
||||
- Successful token issuance with bound certificate.
|
||||
- Request without certificate → `invalid_client`.
|
||||
- Token introspection reveals `cnf.x5t#S256`.
|
||||
- Rotation scenario (old + new cert allowed during grace window).
|
||||
|
||||
## Implementation Checklist
|
||||
|
||||
**DPoP work-stream**
|
||||
1. Extract shared validator into `StellaOps.Auth.Security`; update Scanner references.
|
||||
2. Introduce configuration classes and bind from YAML/environment.
|
||||
3. Implement nonce store (Redis + in-memory), handler integration, and OpenIddict transaction plumbing.
|
||||
4. Stamp `cnf.jkt`, audit events, and metrics; update Mongo documents and migrations.
|
||||
5. Extend docs: `docs/ARCHITECTURE_AUTHORITY.md`, `docs/security/audit-events.md`, `docs/security/rate-limits.md`, CLI/UI references.
|
||||
|
||||
**mTLS work-stream**
|
||||
1. Extend client document/schema and provisioning stores with certificate bindings + sender constraint flag.
|
||||
2. Configure Kestrel/middleware for optional client certificates and validation service.
|
||||
3. Update token issuance/introspection to honour certificate bindings and emit `cnf.x5t#S256`.
|
||||
4. Add auditing/metrics and integration tests (happy path + failure).
|
||||
5. Refresh operator documentation (`docs/ops/authority-backup-restore.md`, `docs/ops/authority-monitoring.md`, sample `authority.yaml`) to cover certificate lifecycle.
|
||||
|
||||
Both streams should conclude with `dotnet test src/StellaOps.Authority.sln` and documentation cross-links so dependent guilds can unblock UI/Signer work.
|
||||
|
||||
## Design Summary
|
||||
- Extract the existing Scanner `DpopProofValidator` stack into a shared `StellaOps.Auth.Security` library used by Authority and resource servers.
|
||||
- Extend Authority configuration (`authority.yaml`) with strongly-typed `senderConstraints.dpop` and `senderConstraints.mtls` sections (map to sample already shown in architecture doc).
|
||||
- Require DPoP proofs on `/token` when the registered client policy is `senderConstraint=dpop`; bind issued access tokens via `cnf.jkt`.
|
||||
- Introduce Authority-managed nonce issuance for “high value” audiences (default: `signer`, `attestor`) with Redis-backed persistence and deterministic auditing.
|
||||
- Enable OAuth 2.0 mTLS (RFC 8705) by storing certificate bindings per client, requesting client certificates at TLS termination, and stamping `cnf.x5t#S256` into issued tokens plus introspection output.
|
||||
- Surface structured logs and counters for both DPoP and mTLS flows; provide integration tests that cover success, replay, invalid proof, and certificate mismatch cases.
|
||||
|
||||
## AUTH-DPOP-11-001 — Proof Validation & Nonce Handling
|
||||
|
||||
**Shared validator**
|
||||
- Move `DpopProofValidator`, option types, and replay cache interfaces from `StellaOps.Scanner.Core` into a new assembly `StellaOps.Auth.Security`.
|
||||
- Provide pluggable caches: `InMemoryDpopReplayCache` (existing) and new `RedisDpopReplayCache` (leveraging the Authority Redis connection).
|
||||
- Ensure the validator exposes the validated `SecurityKey`, `jti`, and `iat` so Authority can construct the `cnf` claim and compute nonce expiry.
|
||||
|
||||
**Configuration model**
|
||||
- Extend `StellaOpsAuthorityOptions.Security` with a `SenderConstraints` property containing:
|
||||
- `Dpop` (`enabled`, `allowedAlgorithms`, `maxAgeSeconds`, `clockSkewSeconds`, `replayWindowSeconds`, `nonce` settings with `enabled`, `ttlSeconds`, `requiredAudiences`, `maxIssuancePerMinute`).
|
||||
- `Mtls` (`enabled`, `requireChainValidation`, `clientCaBundle`, `allowedSubjectPatterns`, `allowedSanTypes`).
|
||||
- Bind from YAML (`authority.security.senderConstraints.*`) while preserving backwards compatibility (defaults keep both disabled).
|
||||
|
||||
**Token endpoint pipeline**
|
||||
- Introduce a scoped OpenIddict handler `ValidateDpopProofHandler` inserted before `ValidateClientCredentialsHandler`.
|
||||
- Determine the required sender constraint from client metadata:
|
||||
- Add `AuthorityClientMetadataKeys.SenderConstraint` storing `dpop` or `mtls`.
|
||||
- Optionally allow per-client overrides for nonce requirement.
|
||||
- When `dpop` is required:
|
||||
- Read the `DPoP` header from the ASP.NET request, reject with `invalid_token` + `WWW-Authenticate: DPoP error="invalid_dpop_proof"` if absent.
|
||||
- Call the shared validator with method/URI. Enforce algorithm allowlist and `iat` window from options.
|
||||
- Persist the `jkt` thumbprint plus replay cache state in the OpenIddict transaction (`AuthorityOpenIddictConstants.DpopKeyThumbprintProperty`, `DpopIssuedAtProperty`).
|
||||
- When the requested audience intersects `SenderConstraints.Dpop.Nonce.RequiredAudiences`, require `nonce` in the proof; on first failure respond with HTTP 401, `error="use_dpop_nonce"`, and include `DPoP-Nonce` header (see nonce note below). Cache the rejection reason for audit logging.
|
||||
|
||||
**Nonce service**
|
||||
- Add `IDpopNonceStore` with methods `IssueAsync(audience, clientId, jkt)` and `TryConsumeAsync(nonce, audience, clientId, jkt)`.
|
||||
- Default implementation `RedisDpopNonceStore` storing SHA-256 hashes of nonces keyed by `audience:clientId:jkt`. TTL comes from `SenderConstraints.Dpop.Nonce.Ttl`.
|
||||
- Create helper `DpopNonceIssuer` used by `ValidateDpopProofHandler` to issue nonces when missing/expired, enforcing issuance rate limits (per options) and tagging audit/log records.
|
||||
- On successful validation (nonce supplied and consumed) stamp metadata into the transaction for auditing.
|
||||
- Update `ClientCredentialsHandlers` to observe nonce enforcement: when a nonce challenge was sent, emit structured audit with `nonce_issued`, `audiences`, and `retry`.
|
||||
|
||||
**Token issuance**
|
||||
- In `HandleClientCredentialsHandler`, if the transaction contains a validated DPoP key:
|
||||
- Build `cnf.jkt` using thumbprint from validator.
|
||||
- Include `auth_time`/`dpop_jti` as needed for diagnostics.
|
||||
- Persist the thumbprint alongside token metadata in Mongo (extend `AuthorityTokenDocument` with `SenderConstraint`, `KeyThumbprint`, `Nonce` fields).
|
||||
|
||||
**Auditing & observability**
|
||||
- Emit new audit events:
|
||||
- `authority.dpop.proof.validated` (success/failure, clientId, audience, thumbprint, nonce status, jti).
|
||||
- `authority.dpop.nonce.issued` and `authority.dpop.nonce.consumed`.
|
||||
- Metrics (Prometheus style):
|
||||
- `authority_dpop_validations_total{result,reason}`.
|
||||
- `authority_dpop_nonce_issued_total{audience}` and `authority_dpop_nonce_fails_total{reason}`.
|
||||
- Structured logs include `authority.sender_constraint=dpop`, `authority.dpop_thumbprint`, `authority.dpop_nonce`.
|
||||
|
||||
**Testing**
|
||||
- Unit tests for the handler pipeline using fake OpenIddict transactions.
|
||||
- Replay/nonce tests with in-memory and Redis stores.
|
||||
- Integration tests in `StellaOps.Authority.Tests` covering:
|
||||
- Valid DPoP proof issuing `cnf.jkt`.
|
||||
- Missing header → challenge with nonce.
|
||||
- Replayed `jti` rejected.
|
||||
- Invalid nonce rejected even after issuance.
|
||||
- Contract tests to ensure `/.well-known/openid-configuration` advertises `dpop_signing_alg_values_supported` and `dpop_nonce_supported` when enabled.
|
||||
|
||||
## AUTH-MTLS-11-002 — Certificate-Bound Tokens
|
||||
|
||||
**Configuration model**
|
||||
- Reuse `SenderConstraints.Mtls` described above; include:
|
||||
- `enforceForAudiences` list (defaults `signer`, `attestor`, `scheduler`).
|
||||
- `certificateRotationGraceSeconds` for overlap.
|
||||
- `allowedClientCertificateAuthorities` absolute paths.
|
||||
|
||||
**Kestrel/TLS pipeline**
|
||||
- Configure Kestrel with `ClientCertificateMode.AllowCertificate` globally and implement middleware that enforces certificate presence only when the resolved client requires mTLS.
|
||||
- Add `IAuthorityClientCertificateValidator` that validates presented certificate chain, SANs (`dns`, `uri`, optional SPIFFE), and thumbprint matches one of the stored bindings.
|
||||
- Cache validation results per connection id to avoid rehashing on every request.
|
||||
|
||||
**Client registration & storage**
|
||||
- Extend `AuthorityClientDocument` with `List<AuthorityClientCertificateBinding>` containing:
|
||||
- `Thumbprint`, `SerialNumber`, `Subject`, `NotBefore`, `NotAfter`, `Sans`, `CreatedAt`, `UpdatedAt`, `Label`.
|
||||
- Provide admin API mutations (`/admin/clients/{id}/certificates`) for ops tooling (deferred implementation but schema ready).
|
||||
- Update plugin provisioning store (`StandardClientProvisioningStore`) to map descriptors with certificate bindings and `senderConstraint`.
|
||||
- Persist binding state in Mongo migrations (index on `{clientId, thumbprint}`).
|
||||
|
||||
**Token issuance & introspection**
|
||||
- Add a transaction property capturing the validated client certificate thumbprint.
|
||||
- `HandleClientCredentialsHandler`:
|
||||
- When mTLS required, ensure certificate info present; reject otherwise.
|
||||
- Stamp `cnf` claim: `principal.SetClaim("cnf", JsonSerializer.Serialize(new { x5t#S256 = thumbprint }))`.
|
||||
- Store binding metadata in issued token document for audit.
|
||||
- Update `ValidateAccessTokenHandler` and introspection responses to surface `cnf.x5t#S256`.
|
||||
- Ensure refresh tokens (if ever enabled) copy the binding data.
|
||||
|
||||
**Auditing & observability**
|
||||
- Audit events:
|
||||
- `authority.mtls.handshake` (success/failure, clientId, thumbprint, issuer, subject).
|
||||
- `authority.mtls.binding.missing` when a required client posts without a cert.
|
||||
- Metrics:
|
||||
- `authority_mtls_handshakes_total{result}`.
|
||||
- `authority_mtls_certificate_rotations_total`.
|
||||
- Logs include `authority.sender_constraint=mtls`, `authority.mtls_thumbprint`, `authority.mtls_subject`.
|
||||
|
||||
**Testing**
|
||||
- Unit tests for certificate validation rules (SAN mismatches, expiry, CA trust).
|
||||
- Integration tests running Kestrel with test certificates:
|
||||
- Successful token issuance with bound certificate.
|
||||
- Request without certificate → `invalid_client`.
|
||||
- Token introspection reveals `cnf.x5t#S256`.
|
||||
- Rotation scenario (old + new cert allowed during grace window).
|
||||
|
||||
## Implementation Checklist
|
||||
|
||||
**DPoP work-stream**
|
||||
1. Extract shared validator into `StellaOps.Auth.Security`; update Scanner references.
|
||||
2. Introduce configuration classes and bind from YAML/environment.
|
||||
3. Implement nonce store (Redis + in-memory), handler integration, and OpenIddict transaction plumbing.
|
||||
4. Stamp `cnf.jkt`, audit events, and metrics; update Mongo documents and migrations.
|
||||
5. Extend docs: `docs/ARCHITECTURE_AUTHORITY.md`, `docs/security/audit-events.md`, `docs/security/rate-limits.md`, CLI/UI references.
|
||||
|
||||
**mTLS work-stream**
|
||||
1. Extend client document/schema and provisioning stores with certificate bindings + sender constraint flag.
|
||||
2. Configure Kestrel/middleware for optional client certificates and validation service.
|
||||
3. Update token issuance/introspection to honour certificate bindings and emit `cnf.x5t#S256`.
|
||||
4. Add auditing/metrics and integration tests (happy path + failure).
|
||||
5. Refresh operator documentation (`docs/ops/authority-backup-restore.md`, `docs/ops/authority-monitoring.md`, sample `authority.yaml`) to cover certificate lifecycle.
|
||||
|
||||
Both streams should conclude with `dotnet test src/Authority/StellaOps.Authority/StellaOps.Authority.sln` and documentation cross-links so dependent guilds can unblock UI/Signer work.
|
||||
|
||||
@@ -2,42 +2,42 @@
|
||||
|
||||
> Created: 2025-10-19 — Plugin Platform Guild & Authority Core
|
||||
> Status: Completed (workshop held 2025-10-20 15:00–16:05 UTC)
|
||||
|
||||
This document tracks preparation, agenda, and outcomes for the scoped-service workshop required before implementing PLUGIN-DI-08-002.
|
||||
|
||||
## Objectives
|
||||
|
||||
- Inventory Authority plug-in surfaces that need scoped service lifetimes.
|
||||
- Confirm session/scope handling for identity-provider registrars and background jobs.
|
||||
- Assign follow-up tasks/actions with owners and due dates.
|
||||
|
||||
## Scheduling Snapshot
|
||||
|
||||
- **Meeting time:** 2025-10-20 15:00–16:00 UTC (10:00–11:00 CDT / 08:00–09:00 PDT).
|
||||
- **Facilitator:** Plugin Platform Guild — Alicia Rivera.
|
||||
- **Attendees (confirmed):** Authority Core — Jasmin Patel; Authority Security Guild — Mohan Singh; Plugin Platform — Alicia Rivera, Leah Chen.
|
||||
- **Optional invitees:** DevOps liaison — Sofia Ortega (accepted).
|
||||
- **Logistics:** Invites sent via shared calendar on 2025-10-19 15:30 UTC with Teams bridge + offline dial-in. Meeting notes will be captured here.
|
||||
- **Preparation deadline:** 2025-10-20 12:00 UTC — complete checklist below.
|
||||
|
||||
## Pre-work Checklist
|
||||
|
||||
- Review `ServiceBindingAttribute` contract introduced by PLUGIN-DI-08-001.
|
||||
- Collect existing Authority plug-in registration code paths to evaluate.
|
||||
- Audit background jobs that assume singleton lifetimes.
|
||||
- Identify plug-in health checks/telemetry surfaces impacted by scoped lifetimes.
|
||||
|
||||
|
||||
This document tracks preparation, agenda, and outcomes for the scoped-service workshop required before implementing PLUGIN-DI-08-002.
|
||||
|
||||
## Objectives
|
||||
|
||||
- Inventory Authority plug-in surfaces that need scoped service lifetimes.
|
||||
- Confirm session/scope handling for identity-provider registrars and background jobs.
|
||||
- Assign follow-up tasks/actions with owners and due dates.
|
||||
|
||||
## Scheduling Snapshot
|
||||
|
||||
- **Meeting time:** 2025-10-20 15:00–16:00 UTC (10:00–11:00 CDT / 08:00–09:00 PDT).
|
||||
- **Facilitator:** Plugin Platform Guild — Alicia Rivera.
|
||||
- **Attendees (confirmed):** Authority Core — Jasmin Patel; Authority Security Guild — Mohan Singh; Plugin Platform — Alicia Rivera, Leah Chen.
|
||||
- **Optional invitees:** DevOps liaison — Sofia Ortega (accepted).
|
||||
- **Logistics:** Invites sent via shared calendar on 2025-10-19 15:30 UTC with Teams bridge + offline dial-in. Meeting notes will be captured here.
|
||||
- **Preparation deadline:** 2025-10-20 12:00 UTC — complete checklist below.
|
||||
|
||||
## Pre-work Checklist
|
||||
|
||||
- Review `ServiceBindingAttribute` contract introduced by PLUGIN-DI-08-001.
|
||||
- Collect existing Authority plug-in registration code paths to evaluate.
|
||||
- Audit background jobs that assume singleton lifetimes.
|
||||
- Identify plug-in health checks/telemetry surfaces impacted by scoped lifetimes.
|
||||
|
||||
### Pre-work References
|
||||
|
||||
| Focus | Path | Notes |
|
||||
|-------|------|-------|
|
||||
| Host DI wiring | `src/StellaOps.Authority/StellaOps.Authority/Program.cs:159` | Startup registers `IAuthorityIdentityProviderRegistry` as a singleton and invokes `AuthorityPluginLoader.RegisterPlugins(...)` before the container is built. Any scoped plugin services will currently be captured in the singleton registry context. |
|
||||
| Registrar discovery | `src/StellaOps.Authority/StellaOps.Authority/Plugins/AuthorityPluginLoader.cs:46` | Loader instantiates `IAuthorityPluginRegistrar` implementations via `Activator.CreateInstance`, so registrars cannot depend on host services yet. Need agreement on whether to move discovery post-build or introduce `ActivatorUtilities`. |
|
||||
| Registry aggregation | `src/StellaOps.Authority/StellaOps.Authority/AuthorityIdentityProviderRegistry.cs:16` | Registry caches `IIdentityProviderPlugin` instances at construction time. With scoped lifetimes we must revisit how providers are resolved (factory vs accessor). |
|
||||
| Standard registrar services | `src/StellaOps.Authority/StellaOps.Authority.Plugin.Standard/StandardPluginRegistrar.cs:21` | All plugin services are registered as singletons today (`StandardUserCredentialStore`, `StandardClientProvisioningStore`, hosted bootstrapper). This registrar is our baseline for migrating to scoped bindings. |
|
||||
| Hosted bootstrapper | `src/StellaOps.Authority/StellaOps.Authority.Plugin.Standard/Bootstrap/StandardPluginBootstrapper.cs:17` | Background job directly consumes `StandardUserCredentialStore`. If the store becomes scoped we will need an `IServiceScopeFactory` bridge. |
|
||||
| Password grant handler | `src/StellaOps.Authority/StellaOps.Authority/OpenIddict/Handlers/PasswordGrantHandlers.cs:26` | Password flow resolves `IIdentityProviderPlugin` during scoped requests. Scope semantics must ensure credential stores stay cancellation-aware. |
|
||||
| Client credential handler | `src/StellaOps.Authority/StellaOps.Authority/OpenIddict/Handlers/ClientCredentialsHandlers.cs:21` | Handler fetches provider + `ClientProvisioning` store; confirms need for consistent scoping in both user and client flows. |
|
||||
| Host DI wiring | `src/Authority/StellaOps.Authority/StellaOps.Authority/Program.cs:159` | Startup registers `IAuthorityIdentityProviderRegistry` as a singleton and invokes `AuthorityPluginLoader.RegisterPlugins(...)` before the container is built. Any scoped plugin services will currently be captured in the singleton registry context. |
|
||||
| Registrar discovery | `src/Authority/StellaOps.Authority/StellaOps.Authority/Plugins/AuthorityPluginLoader.cs:46` | Loader instantiates `IAuthorityPluginRegistrar` implementations via `Activator.CreateInstance`, so registrars cannot depend on host services yet. Need agreement on whether to move discovery post-build or introduce `ActivatorUtilities`. |
|
||||
| Registry aggregation | `src/Authority/StellaOps.Authority/StellaOps.Authority/AuthorityIdentityProviderRegistry.cs:16` | Registry caches `IIdentityProviderPlugin` instances at construction time. With scoped lifetimes we must revisit how providers are resolved (factory vs accessor). |
|
||||
| Standard registrar services | `src/Authority/StellaOps.Authority/StellaOps.Authority.Plugin.Standard/StandardPluginRegistrar.cs:21` | All plugin services are registered as singletons today (`StandardUserCredentialStore`, `StandardClientProvisioningStore`, hosted bootstrapper). This registrar is our baseline for migrating to scoped bindings. |
|
||||
| Hosted bootstrapper | `src/Authority/StellaOps.Authority/StellaOps.Authority.Plugin.Standard/Bootstrap/StandardPluginBootstrapper.cs:17` | Background job directly consumes `StandardUserCredentialStore`. If the store becomes scoped we will need an `IServiceScopeFactory` bridge. |
|
||||
| Password grant handler | `src/Authority/StellaOps.Authority/StellaOps.Authority/OpenIddict/Handlers/PasswordGrantHandlers.cs:26` | Password flow resolves `IIdentityProviderPlugin` during scoped requests. Scope semantics must ensure credential stores stay cancellation-aware. |
|
||||
| Client credential handler | `src/Authority/StellaOps.Authority/StellaOps.Authority/OpenIddict/Handlers/ClientCredentialsHandlers.cs:21` | Handler fetches provider + `ClientProvisioning` store; confirms need for consistent scoping in both user and client flows. |
|
||||
|
||||
## Preliminary Findings — 2025-10-20
|
||||
|
||||
@@ -49,15 +49,15 @@ This document tracks preparation, agenda, and outcomes for the scoped-service wo
|
||||
- 2025-10-20 (PLUGIN-DI-08-003): Registry implementation updated to expose metadata + scoped handles; OpenIddict flows, bootstrap endpoints, and `/health` now resolve providers via scoped leases with accompanying test coverage.
|
||||
- 2025-10-20 (PLUGIN-DI-08-004): Authority plugin loader now instantiates registrars via scoped DI activations and honours `[ServiceBinding]` metadata in plugin assemblies.
|
||||
- 2025-10-20 (PLUGIN-DI-08-005): `StandardPluginBootstrapper` shifted to scope-per-run execution using `IServiceScopeFactory`, enabling future scoped stores without singleton leaks.
|
||||
|
||||
## Draft Agenda
|
||||
|
||||
1. Context recap (5 min) — why scoped DI is needed; summary of PLUGIN-DI-08-001 changes.
|
||||
2. Authority plug-in surfaces (15 min) — registrars, background services, telemetry.
|
||||
3. Session handling strategy (10 min) — scope creation semantics, cancellation propagation.
|
||||
4. Action items & owners (10 min) — capture code/docs/test tasks with due dates.
|
||||
5. Risks & follow-ups (5 min) — dependencies, rollout sequencing.
|
||||
|
||||
|
||||
## Draft Agenda
|
||||
|
||||
1. Context recap (5 min) — why scoped DI is needed; summary of PLUGIN-DI-08-001 changes.
|
||||
2. Authority plug-in surfaces (15 min) — registrars, background services, telemetry.
|
||||
3. Session handling strategy (10 min) — scope creation semantics, cancellation propagation.
|
||||
4. Action items & owners (10 min) — capture code/docs/test tasks with due dates.
|
||||
5. Risks & follow-ups (5 min) — dependencies, rollout sequencing.
|
||||
|
||||
## Notes
|
||||
|
||||
- Session opened with recap of scoped-service goals and PLUGIN-DI-08-001 changes, confirming Authority readiness to adopt `[ServiceBinding]` metadata.
|
||||
@@ -65,11 +65,11 @@ This document tracks preparation, agenda, and outcomes for the scoped-service wo
|
||||
- Standard plug-in bootstrap will create scopes via `IServiceScopeFactory` and pass cancellation tokens through to avoid lingering singleton references.
|
||||
- Authority Plugin Loader will enumerate plug-in assemblies at startup but defer registrar activation until a scoped service provider is available, aligning with PLUGIN-DI-08-004 implementation.
|
||||
- Follow-up engineering tasks assigned to land PLUGIN-DI-08-002 code path adjustments and Authority host updates before 2025-10-24.
|
||||
|
||||
## Action Item Log
|
||||
|
||||
| Item | Owner | Due | Status | Notes |
|
||||
|------|-------|-----|--------|-------|
|
||||
|
||||
## Action Item Log
|
||||
|
||||
| Item | Owner | Due | Status | Notes |
|
||||
|------|-------|-----|--------|-------|
|
||||
| Confirm meeting time | Alicia Rivera | 2025-10-19 15:30 UTC | DONE | Calendar invite sent; all required attendees accepted |
|
||||
| Compile Authority plug-in DI entry points | Jasmin Patel | 2025-10-20 | DONE (2025-10-20) | Scoped-service touchpoints summarised in **Pre-work References** and **Preliminary Findings** ahead of the workshop. |
|
||||
| Outline scoped-session pattern for background jobs | Leah Chen | 2025-10-21 | DONE (2025-10-20) | Pattern agreed: bootstrap services must open transient scopes per execution via `IServiceScopeFactory`; document update to follow in PLUGIN-DI-08-002 patch. |
|
||||
|
||||
@@ -7,39 +7,39 @@ fixture sets, where they live, and how to regenerate them safely.
|
||||
|
||||
## GHSA ↔ OSV parity fixtures
|
||||
|
||||
- **Location:** `src/StellaOps.Concelier.Connector.Osv.Tests/Fixtures/osv-ghsa.*.json`
|
||||
- **Location:** `src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Osv.Tests/Fixtures/osv-ghsa.*.json`
|
||||
- **Purpose:** Exercised by `OsvGhsaParityRegressionTests` to ensure OSV + GHSA outputs stay aligned on aliases,
|
||||
ranges, references, and credits.
|
||||
- **Regeneration:** Either run the test harness with online regeneration (`UPDATE_PARITY_FIXTURES=1 dotnet test src/StellaOps.Concelier.Connector.Osv.Tests/StellaOps.Concelier.Connector.Osv.Tests.csproj`)
|
||||
- **Regeneration:** Either run the test harness with online regeneration (`UPDATE_PARITY_FIXTURES=1 dotnet test src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Osv.Tests/StellaOps.Concelier.Connector.Osv.Tests.csproj`)
|
||||
or execute the fixture updater (`dotnet run --project tools/FixtureUpdater/FixtureUpdater.csproj`). Both paths
|
||||
normalise timestamps and canonical ordering.
|
||||
- **SemVer provenance:** The regenerated fixtures should show `normalizedVersions[].notes` in the
|
||||
`osv:{ecosystem}:{advisoryId}:{identifier}` shape emitted by `SemVerRangeRuleBuilder`. Confirm the
|
||||
constraints and notes line up with GHSA/NVD composites before committing.
|
||||
- **Verification:** Inspect the diff, then re-run `dotnet test src/StellaOps.Concelier.Connector.Osv.Tests/StellaOps.Concelier.Connector.Osv.Tests.csproj` to confirm parity.
|
||||
- **Verification:** Inspect the diff, then re-run `dotnet test src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Osv.Tests/StellaOps.Concelier.Connector.Osv.Tests.csproj` to confirm parity.
|
||||
|
||||
## GHSA credit parity fixtures
|
||||
|
||||
- **Location:** `src/StellaOps.Concelier.Connector.Ghsa.Tests/Fixtures/credit-parity.{ghsa,osv,nvd}.json`
|
||||
- **Location:** `src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Ghsa.Tests/Fixtures/credit-parity.{ghsa,osv,nvd}.json`
|
||||
- **Purpose:** Exercised by `GhsaCreditParityRegressionTests` to guarantee GHSA/NVD/OSV acknowledgements remain in lockstep.
|
||||
- **Regeneration:** `dotnet run --project tools/FixtureUpdater/FixtureUpdater.csproj` rewrites all three canonical snapshots.
|
||||
- **Verification:** `dotnet test src/StellaOps.Concelier.Connector.Ghsa.Tests/StellaOps.Concelier.Connector.Ghsa.Tests.csproj`.
|
||||
- **Verification:** `dotnet test src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Ghsa.Tests/StellaOps.Concelier.Connector.Ghsa.Tests.csproj`.
|
||||
|
||||
> Always commit fixture changes together with the code that motivated them and reference the regression test that guards the behaviour.
|
||||
|
||||
## Apple security update fixtures
|
||||
|
||||
- **Location:** `src/StellaOps.Concelier.Connector.Vndr.Apple.Tests/Apple/Fixtures/*.html` and `.expected.json`.
|
||||
- **Location:** `src/Concelier/__Tests/StellaOps.Concelier.Connector.Vndr.Apple.Tests/Apple/Fixtures/*.html` and `.expected.json`.
|
||||
- **Purpose:** Exercised by `AppleLiveRegressionTests` to guarantee the Apple HTML parser and mapper stay deterministic while covering Rapid Security Responses and multi-device advisories.
|
||||
- **Regeneration:** Use the helper scripts (`scripts/update-apple-fixtures.sh` or `scripts/update-apple-fixtures.ps1`). They export `UPDATE_APPLE_FIXTURES=1`, propagate the flag through `WSLENV`, touch `.update-apple-fixtures`, and then run the Apple test project. This keeps WSL/VSCode test invocations in sync while the refresh workflow fetches live Apple support pages, sanitises them, and rewrites both the HTML and expected DTO snapshots with normalised ordering.
|
||||
- **Verification:** Inspect the generated diffs and re-run `dotnet test src/StellaOps.Concelier.Connector.Vndr.Apple.Tests/StellaOps.Concelier.Connector.Vndr.Apple.Tests.csproj` without the env var to confirm determinism.
|
||||
- **Verification:** Inspect the generated diffs and re-run `dotnet test src/Concelier/__Tests/StellaOps.Concelier.Connector.Vndr.Apple.Tests/StellaOps.Concelier.Connector.Vndr.Apple.Tests.csproj` without the env var to confirm determinism.
|
||||
|
||||
> **Tip for other connector owners:** mirror the sentinel + `WSLENV` pattern (`touch .update-<connector>-fixtures`, append the env var via `WSLENV`) when you add fixture refresh scripts so contributors running under WSL inherit the regeneration flag automatically.
|
||||
|
||||
## KISA advisory fixtures
|
||||
|
||||
- **Location:** `src/StellaOps.Concelier.Connector.Kisa.Tests/Fixtures/kisa-{feed,detail}.(xml|json)`
|
||||
- **Location:** `src/Concelier/__Tests/StellaOps.Concelier.Connector.Kisa.Tests/Fixtures/kisa-{feed,detail}.(xml|json)`
|
||||
- **Purpose:** Used by `KisaConnectorTests` to verify Hangul-aware fetch → parse → map flows and to assert telemetry counters stay wired.
|
||||
- **Regeneration:** `UPDATE_KISA_FIXTURES=1 dotnet test src/StellaOps.Concelier.Connector.Kisa.Tests/StellaOps.Concelier.Connector.Kisa.Tests.csproj`
|
||||
- **Regeneration:** `UPDATE_KISA_FIXTURES=1 dotnet test src/Concelier/__Tests/StellaOps.Concelier.Connector.Kisa.Tests/StellaOps.Concelier.Connector.Kisa.Tests.csproj`
|
||||
- **Verification:** Re-run the same test suite without the env var; confirm advisory content remains NFC-normalised and HTML is sanitised. Metrics assertions will fail if counters drift.
|
||||
- **Localisation note:** RSS `category` values (e.g. `취약점정보`) remain in Hangul—do not translate them in fixtures; they feed directly into metrics/log tags.
|
||||
|
||||
@@ -39,7 +39,7 @@ The messages use structured properties (`Idx`, `Category`, `DocumentId`, `Severi
|
||||
- Hangul fields (`title`, `summary`, `category`, `reference.label`, product vendor/name) are normalised to NFC before storage. Sample category `취약점정보` roughly translates to “vulnerability information”.
|
||||
- Advisory HTML is sanitised via `HtmlContentSanitizer`, stripping script/style while preserving inline anchors for translation pipelines.
|
||||
- Metrics carry Hangul `category` tags and logging keeps Hangul strings intact; this ensures air-gapped operators can validate native-language content without relying on MT.
|
||||
- Fixtures live under `src/StellaOps.Concelier.Connector.Kisa.Tests/Fixtures/`. Regenerate with `UPDATE_KISA_FIXTURES=1 dotnet test src/StellaOps.Concelier.Connector.Kisa.Tests/StellaOps.Concelier.Connector.Kisa.Tests.csproj`.
|
||||
- Fixtures live under `src/Concelier/__Tests/StellaOps.Concelier.Connector.Kisa.Tests/Fixtures/`. Regenerate with `UPDATE_KISA_FIXTURES=1 dotnet test src/Concelier/__Tests/StellaOps.Concelier.Connector.Kisa.Tests/StellaOps.Concelier.Connector.Kisa.Tests.csproj`.
|
||||
- The regression suite asserts canonical mapping, state cleanup, and telemetry counters (`KisaConnectorTests.Telemetry_RecordsMetrics`) so QA can track instrumentation drift.
|
||||
|
||||
For operator docs, link to this brief when documenting Hangul handling or counter dashboards so localisation reviewers have a single reference point.
|
||||
|
||||
@@ -1,154 +1,154 @@
|
||||
# Concelier SemVer Merge Playbook (Sprint 1–2)
|
||||
|
||||
This playbook describes how the merge layer and connector teams should emit the new SemVer primitives introduced in Sprint 1–2, how those primitives become normalized version rules, and how downstream jobs query them deterministically.
|
||||
|
||||
## 1. What landed in Sprint 1–2
|
||||
|
||||
- `RangePrimitives.SemVer` now infers a canonical `style` (`range`, `exact`, `lt`, `lte`, `gt`, `gte`) and captures `exactValue` when the constraint is a single version.
|
||||
- `NormalizedVersionRule` documents the analytics-friendly projection of each `AffectedPackage` coverage entry and is persisted alongside legacy `versionRanges`.
|
||||
- `AdvisoryProvenance.decisionReason` records whether merge resolution favored precedence, freshness, or a tie-breaker comparison.
|
||||
|
||||
See `src/StellaOps.Concelier.Models/CANONICAL_RECORDS.md` for the full schema and field descriptions.
|
||||
|
||||
## 2. Mapper pattern
|
||||
|
||||
Connectors should emit SemVer primitives as soon as they can normalize a vendor constraint. The helper `SemVerPrimitiveExtensions.ToNormalizedVersionRule` turns those primitives into the persisted rules:
|
||||
|
||||
```csharp
|
||||
var primitive = new SemVerPrimitive(
|
||||
introduced: "1.2.3",
|
||||
introducedInclusive: true,
|
||||
fixed: "2.0.0",
|
||||
fixedInclusive: false,
|
||||
lastAffected: null,
|
||||
lastAffectedInclusive: false,
|
||||
constraintExpression: ">=1.2.3 <2.0.0",
|
||||
exactValue: null);
|
||||
|
||||
var rule = primitive.ToNormalizedVersionRule(notes: "nvd:CVE-2025-1234");
|
||||
// rule => scheme=semver, type=range, min=1.2.3, minInclusive=true, max=2.0.0, maxInclusive=false
|
||||
```
|
||||
|
||||
If you omit the optional `notes` argument, `ToNormalizedVersionRule` now falls back to the primitive’s `ConstraintExpression`, ensuring the original comparator expression is preserved for provenance/audit queries.
|
||||
|
||||
Emit the resulting rule inside `AffectedPackage.NormalizedVersions` while continuing to populate `AffectedVersionRange.RangeExpression` for backward compatibility.
|
||||
|
||||
## 3. Merge dedupe flow
|
||||
|
||||
During merge, feed all package candidates through `NormalizedVersionRuleComparer.Instance` prior to persistence. The comparer orders by scheme → type → min → minInclusive → max → maxInclusive → value → notes, guaranteeing consistent document layout and making `$unwind` pipelines deterministic.
|
||||
|
||||
If multiple connectors emit identical constraints, the merge layer should:
|
||||
|
||||
1. Combine provenance entries (preserving one per source).
|
||||
2. Preserve a single normalized rule instance (thanks to `NormalizedVersionRuleEqualityComparer.Instance`).
|
||||
3. Attach `decisionReason="precedence"` if one source overrides another.
|
||||
|
||||
## 4. Example Mongo pipeline
|
||||
|
||||
Use the following aggregation to locate advisories that affect a specific SemVer:
|
||||
|
||||
```javascript
|
||||
db.advisories.aggregate([
|
||||
{ $match: { "affectedPackages.type": "semver", "affectedPackages.identifier": "pkg:npm/lodash" } },
|
||||
{ $unwind: "$affectedPackages" },
|
||||
{ $unwind: "$affectedPackages.normalizedVersions" },
|
||||
{ $match: {
|
||||
$or: [
|
||||
{ "affectedPackages.normalizedVersions.type": "exact",
|
||||
"affectedPackages.normalizedVersions.value": "4.17.21" },
|
||||
{ "affectedPackages.normalizedVersions.type": "range",
|
||||
"affectedPackages.normalizedVersions.min": { $lte: "4.17.21" },
|
||||
"affectedPackages.normalizedVersions.max": { $gt: "4.17.21" } },
|
||||
{ "affectedPackages.normalizedVersions.type": "gte",
|
||||
"affectedPackages.normalizedVersions.min": { $lte: "4.17.21" } },
|
||||
{ "affectedPackages.normalizedVersions.type": "lte",
|
||||
"affectedPackages.normalizedVersions.max": { $gte: "4.17.21" } }
|
||||
]
|
||||
}},
|
||||
{ $project: { advisoryKey: 1, title: 1, "affectedPackages.identifier": 1 } }
|
||||
]);
|
||||
```
|
||||
|
||||
Pair this query with the indexes listed in [Normalized Versions Query Guide](mongo_indices.md).
|
||||
|
||||
## 5. Recommended indexes
|
||||
|
||||
| Collection | Index | Purpose |
|
||||
|------------|-------|---------|
|
||||
| `advisory` | `{ "affectedPackages.identifier": 1, "affectedPackages.normalizedVersions.scheme": 1, "affectedPackages.normalizedVersions.type": 1 }` (compound, multikey) | Speeds up `$match` on identifier + rule style. |
|
||||
| `advisory` | `{ "affectedPackages.normalizedVersions.value": 1 }` (sparse) | Optimizes lookups for exact version hits. |
|
||||
|
||||
Coordinate with the Storage team when enabling these indexes so deployment windows account for collection size.
|
||||
|
||||
## 6. Dual-write rollout
|
||||
|
||||
Follow the operational checklist in `docs/ops/migrations/SEMVER_STYLE.md`. The summary:
|
||||
|
||||
1. **Dual write (now)** – emit both legacy `versionRanges` and the new `normalizedVersions`.
|
||||
2. **Backfill** – follow the storage migration in `docs/ops/migrations/SEMVER_STYLE.md` to rewrite historical advisories before switching consumers.
|
||||
3. **Verify** – run the aggregation above (with `explain("executionStats")`) to ensure the new indexes are used.
|
||||
4. **Cutover** – after consumers switch to normalized rules, mark the old `rangeExpression` as deprecated.
|
||||
|
||||
## 7. Checklist for connectors & merge
|
||||
|
||||
- [ ] Populate `SemVerPrimitive` for every SemVer-friendly constraint.
|
||||
- [ ] Call `ToNormalizedVersionRule` and store the result.
|
||||
- [ ] Emit provenance masks covering both `versionRanges[].primitives.semver` and `normalizedVersions[]`.
|
||||
- [ ] Ensure merge deduping relies on the canonical comparer.
|
||||
- [ ] Capture merge decisions via `decisionReason`.
|
||||
- [ ] Confirm integration tests include fixtures with normalized rules and SemVer styles.
|
||||
|
||||
For deeper query examples and maintenance tasks, continue with [Normalized Versions Query Guide](mongo_indices.md).
|
||||
|
||||
## 8. Storage projection reference
|
||||
|
||||
`NormalizedVersionDocumentFactory` copies each normalized rule into MongoDB using the shape below. Use this as a contract when reviewing connector fixtures or diagnosing merge/storage diffs:
|
||||
|
||||
```json
|
||||
{
|
||||
"packageId": "pkg:npm/example",
|
||||
"packageType": "npm",
|
||||
"scheme": "semver",
|
||||
"type": "range",
|
||||
"style": "range",
|
||||
"min": "1.2.3",
|
||||
"minInclusive": true,
|
||||
"max": "2.0.0",
|
||||
"maxInclusive": false,
|
||||
"value": null,
|
||||
"notes": "ghsa:GHSA-xxxx-yyyy",
|
||||
"decisionReason": "ghsa-precedence-over-nvd",
|
||||
"constraint": ">= 1.2.3 < 2.0.0",
|
||||
"source": "ghsa",
|
||||
"recordedAt": "2025-10-11T00:00:00Z"
|
||||
}
|
||||
```
|
||||
|
||||
For distro-specific ranges (`nevra`, `evr`) the same envelope applies with `scheme` switched accordingly. Example:
|
||||
|
||||
```json
|
||||
{
|
||||
"packageId": "bash",
|
||||
"packageType": "rpm",
|
||||
"scheme": "nevra",
|
||||
"type": "range",
|
||||
"style": "range",
|
||||
"min": "0:4.4.18-2.el7",
|
||||
"minInclusive": true,
|
||||
"max": "0:4.4.20-1.el7",
|
||||
"maxInclusive": false,
|
||||
"value": null,
|
||||
"notes": "redhat:RHSA-2025:1234",
|
||||
"decisionReason": "rhel-priority-over-nvd",
|
||||
"constraint": "<= 0:4.4.20-1.el7",
|
||||
"source": "redhat",
|
||||
"recordedAt": "2025-10-11T00:00:00Z"
|
||||
}
|
||||
```
|
||||
|
||||
If a new scheme is required (for example, `apple.build` or `ios.semver`), raise it with the Models team before emitting documents so merge comparers and hashing logic can incorporate the change deterministically.
|
||||
|
||||
## 9. Observability signals
|
||||
|
||||
- `concelier.merge.normalized_rules` (counter, tags: `package_type`, `scheme`) – increments once per normalized rule retained after precedence merge.
|
||||
- `concelier.merge.normalized_rules_missing` (counter, tags: `package_type`) – increments when a merged package still carries version ranges but no normalized rules; watch for spikes to catch connectors that have not emitted normalized arrays yet.
|
||||
# Concelier SemVer Merge Playbook (Sprint 1–2)
|
||||
|
||||
This playbook describes how the merge layer and connector teams should emit the new SemVer primitives introduced in Sprint 1–2, how those primitives become normalized version rules, and how downstream jobs query them deterministically.
|
||||
|
||||
## 1. What landed in Sprint 1–2
|
||||
|
||||
- `RangePrimitives.SemVer` now infers a canonical `style` (`range`, `exact`, `lt`, `lte`, `gt`, `gte`) and captures `exactValue` when the constraint is a single version.
|
||||
- `NormalizedVersionRule` documents the analytics-friendly projection of each `AffectedPackage` coverage entry and is persisted alongside legacy `versionRanges`.
|
||||
- `AdvisoryProvenance.decisionReason` records whether merge resolution favored precedence, freshness, or a tie-breaker comparison.
|
||||
|
||||
See `src/Concelier/__Libraries/StellaOps.Concelier.Models/CANONICAL_RECORDS.md` for the full schema and field descriptions.
|
||||
|
||||
## 2. Mapper pattern
|
||||
|
||||
Connectors should emit SemVer primitives as soon as they can normalize a vendor constraint. The helper `SemVerPrimitiveExtensions.ToNormalizedVersionRule` turns those primitives into the persisted rules:
|
||||
|
||||
```csharp
|
||||
var primitive = new SemVerPrimitive(
|
||||
introduced: "1.2.3",
|
||||
introducedInclusive: true,
|
||||
fixed: "2.0.0",
|
||||
fixedInclusive: false,
|
||||
lastAffected: null,
|
||||
lastAffectedInclusive: false,
|
||||
constraintExpression: ">=1.2.3 <2.0.0",
|
||||
exactValue: null);
|
||||
|
||||
var rule = primitive.ToNormalizedVersionRule(notes: "nvd:CVE-2025-1234");
|
||||
// rule => scheme=semver, type=range, min=1.2.3, minInclusive=true, max=2.0.0, maxInclusive=false
|
||||
```
|
||||
|
||||
If you omit the optional `notes` argument, `ToNormalizedVersionRule` now falls back to the primitive’s `ConstraintExpression`, ensuring the original comparator expression is preserved for provenance/audit queries.
|
||||
|
||||
Emit the resulting rule inside `AffectedPackage.NormalizedVersions` while continuing to populate `AffectedVersionRange.RangeExpression` for backward compatibility.
|
||||
|
||||
## 3. Merge dedupe flow
|
||||
|
||||
During merge, feed all package candidates through `NormalizedVersionRuleComparer.Instance` prior to persistence. The comparer orders by scheme → type → min → minInclusive → max → maxInclusive → value → notes, guaranteeing consistent document layout and making `$unwind` pipelines deterministic.
|
||||
|
||||
If multiple connectors emit identical constraints, the merge layer should:
|
||||
|
||||
1. Combine provenance entries (preserving one per source).
|
||||
2. Preserve a single normalized rule instance (thanks to `NormalizedVersionRuleEqualityComparer.Instance`).
|
||||
3. Attach `decisionReason="precedence"` if one source overrides another.
|
||||
|
||||
## 4. Example Mongo pipeline
|
||||
|
||||
Use the following aggregation to locate advisories that affect a specific SemVer:
|
||||
|
||||
```javascript
|
||||
db.advisories.aggregate([
|
||||
{ $match: { "affectedPackages.type": "semver", "affectedPackages.identifier": "pkg:npm/lodash" } },
|
||||
{ $unwind: "$affectedPackages" },
|
||||
{ $unwind: "$affectedPackages.normalizedVersions" },
|
||||
{ $match: {
|
||||
$or: [
|
||||
{ "affectedPackages.normalizedVersions.type": "exact",
|
||||
"affectedPackages.normalizedVersions.value": "4.17.21" },
|
||||
{ "affectedPackages.normalizedVersions.type": "range",
|
||||
"affectedPackages.normalizedVersions.min": { $lte: "4.17.21" },
|
||||
"affectedPackages.normalizedVersions.max": { $gt: "4.17.21" } },
|
||||
{ "affectedPackages.normalizedVersions.type": "gte",
|
||||
"affectedPackages.normalizedVersions.min": { $lte: "4.17.21" } },
|
||||
{ "affectedPackages.normalizedVersions.type": "lte",
|
||||
"affectedPackages.normalizedVersions.max": { $gte: "4.17.21" } }
|
||||
]
|
||||
}},
|
||||
{ $project: { advisoryKey: 1, title: 1, "affectedPackages.identifier": 1 } }
|
||||
]);
|
||||
```
|
||||
|
||||
Pair this query with the indexes listed in [Normalized Versions Query Guide](mongo_indices.md).
|
||||
|
||||
## 5. Recommended indexes
|
||||
|
||||
| Collection | Index | Purpose |
|
||||
|------------|-------|---------|
|
||||
| `advisory` | `{ "affectedPackages.identifier": 1, "affectedPackages.normalizedVersions.scheme": 1, "affectedPackages.normalizedVersions.type": 1 }` (compound, multikey) | Speeds up `$match` on identifier + rule style. |
|
||||
| `advisory` | `{ "affectedPackages.normalizedVersions.value": 1 }` (sparse) | Optimizes lookups for exact version hits. |
|
||||
|
||||
Coordinate with the Storage team when enabling these indexes so deployment windows account for collection size.
|
||||
|
||||
## 6. Dual-write rollout
|
||||
|
||||
Follow the operational checklist in `docs/ops/migrations/SEMVER_STYLE.md`. The summary:
|
||||
|
||||
1. **Dual write (now)** – emit both legacy `versionRanges` and the new `normalizedVersions`.
|
||||
2. **Backfill** – follow the storage migration in `docs/ops/migrations/SEMVER_STYLE.md` to rewrite historical advisories before switching consumers.
|
||||
3. **Verify** – run the aggregation above (with `explain("executionStats")`) to ensure the new indexes are used.
|
||||
4. **Cutover** – after consumers switch to normalized rules, mark the old `rangeExpression` as deprecated.
|
||||
|
||||
## 7. Checklist for connectors & merge
|
||||
|
||||
- [ ] Populate `SemVerPrimitive` for every SemVer-friendly constraint.
|
||||
- [ ] Call `ToNormalizedVersionRule` and store the result.
|
||||
- [ ] Emit provenance masks covering both `versionRanges[].primitives.semver` and `normalizedVersions[]`.
|
||||
- [ ] Ensure merge deduping relies on the canonical comparer.
|
||||
- [ ] Capture merge decisions via `decisionReason`.
|
||||
- [ ] Confirm integration tests include fixtures with normalized rules and SemVer styles.
|
||||
|
||||
For deeper query examples and maintenance tasks, continue with [Normalized Versions Query Guide](mongo_indices.md).
|
||||
|
||||
## 8. Storage projection reference
|
||||
|
||||
`NormalizedVersionDocumentFactory` copies each normalized rule into MongoDB using the shape below. Use this as a contract when reviewing connector fixtures or diagnosing merge/storage diffs:
|
||||
|
||||
```json
|
||||
{
|
||||
"packageId": "pkg:npm/example",
|
||||
"packageType": "npm",
|
||||
"scheme": "semver",
|
||||
"type": "range",
|
||||
"style": "range",
|
||||
"min": "1.2.3",
|
||||
"minInclusive": true,
|
||||
"max": "2.0.0",
|
||||
"maxInclusive": false,
|
||||
"value": null,
|
||||
"notes": "ghsa:GHSA-xxxx-yyyy",
|
||||
"decisionReason": "ghsa-precedence-over-nvd",
|
||||
"constraint": ">= 1.2.3 < 2.0.0",
|
||||
"source": "ghsa",
|
||||
"recordedAt": "2025-10-11T00:00:00Z"
|
||||
}
|
||||
```
|
||||
|
||||
For distro-specific ranges (`nevra`, `evr`) the same envelope applies with `scheme` switched accordingly. Example:
|
||||
|
||||
```json
|
||||
{
|
||||
"packageId": "bash",
|
||||
"packageType": "rpm",
|
||||
"scheme": "nevra",
|
||||
"type": "range",
|
||||
"style": "range",
|
||||
"min": "0:4.4.18-2.el7",
|
||||
"minInclusive": true,
|
||||
"max": "0:4.4.20-1.el7",
|
||||
"maxInclusive": false,
|
||||
"value": null,
|
||||
"notes": "redhat:RHSA-2025:1234",
|
||||
"decisionReason": "rhel-priority-over-nvd",
|
||||
"constraint": "<= 0:4.4.20-1.el7",
|
||||
"source": "redhat",
|
||||
"recordedAt": "2025-10-11T00:00:00Z"
|
||||
}
|
||||
```
|
||||
|
||||
If a new scheme is required (for example, `apple.build` or `ios.semver`), raise it with the Models team before emitting documents so merge comparers and hashing logic can incorporate the change deterministically.
|
||||
|
||||
## 9. Observability signals
|
||||
|
||||
- `concelier.merge.normalized_rules` (counter, tags: `package_type`, `scheme`) – increments once per normalized rule retained after precedence merge.
|
||||
- `concelier.merge.normalized_rules_missing` (counter, tags: `package_type`) – increments when a merged package still carries version ranges but no normalized rules; watch for spikes to catch connectors that have not emitted normalized arrays yet.
|
||||
|
||||
@@ -4,7 +4,7 @@ _Status date: 2025-10-20 19:10 UTC_
|
||||
|
||||
This dashboard tracks connector readiness for emitting `AffectedPackage.NormalizedVersions` arrays and highlights upcoming coordination checkpoints. Use it alongside:
|
||||
|
||||
- [`src/StellaOps.Concelier.Merge/RANGE_PRIMITIVES_COORDINATION.md`](../../src/StellaOps.Concelier.Merge/RANGE_PRIMITIVES_COORDINATION.md) for detailed guidance and timelines.
|
||||
- [`src/Concelier/__Libraries/StellaOps.Concelier.Merge/RANGE_PRIMITIVES_COORDINATION.md`](../../src/Concelier/__Libraries/StellaOps.Concelier.Merge/RANGE_PRIMITIVES_COORDINATION.md) for detailed guidance and timelines.
|
||||
- [Concelier SemVer Merge Playbook](merge_semver_playbook.md) §8 for persisted Mongo document shapes.
|
||||
- [Normalized Versions Query Guide](mongo_indices.md) for index/query validation steps.
|
||||
|
||||
@@ -20,20 +20,20 @@ This dashboard tracks connector readiness for emitting `AffectedPackage.Normaliz
|
||||
|
||||
| Connector | Owner team | Normalized versions status | Last update | Next action / link |
|
||||
|-----------|------------|---------------------------|-------------|--------------------|
|
||||
| Acsc | BE-Conn-ACSC | ❌ Not started – normalized helper pending relay stability | 2025-10-20 | Prepare builder integration plan for 2025-10-24 kickoff; update `src/StellaOps.Concelier.Connector.Acsc/TASKS.md` once branch opens. |
|
||||
| Cccs | BE-Conn-CCCS | ⚠️ DOING – trailing-version helper MR reviewing (due 2025-10-21) | 2025-10-20 | Land helper + fixture refresh, post merge-counter screenshot; `src/StellaOps.Concelier.Connector.Cccs/TASKS.md`. |
|
||||
| CertBund | BE-Conn-CERTBUND | ⚠️ In progress – localisation translator WIP (due 2025-10-22) | 2025-10-20 | Finish translator + provenance notes, regenerate fixtures; `src/StellaOps.Concelier.Connector.CertBund/TASKS.md`. |
|
||||
| Acsc | BE-Conn-ACSC | ❌ Not started – normalized helper pending relay stability | 2025-10-20 | Prepare builder integration plan for 2025-10-24 kickoff; update `src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Acsc/TASKS.md` once branch opens. |
|
||||
| Cccs | BE-Conn-CCCS | ⚠️ DOING – trailing-version helper MR reviewing (due 2025-10-21) | 2025-10-20 | Land helper + fixture refresh, post merge-counter screenshot; `src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Cccs/TASKS.md`. |
|
||||
| CertBund | BE-Conn-CERTBUND | ⚠️ In progress – localisation translator WIP (due 2025-10-22) | 2025-10-20 | Finish translator + provenance notes, regenerate fixtures; `src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.CertBund/TASKS.md`. |
|
||||
| CertCc | BE-Conn-CERTCC | ✅ Complete – `certcc.vendor` rules emitting | 2025-10-20 | Monitor VINCE payload changes; no action. |
|
||||
| Kev | BE-Conn-KEV | ✅ Complete – catalog/due-date rules verified | 2025-10-20 | Routine monitoring only. |
|
||||
| Cve | BE-Conn-CVE | ✅ Complete – SemVer normalized rules live | 2025-10-20 | Keep fixtures in sync as CVE schema evolves. |
|
||||
| Ghsa | BE-Conn-GHSA | ✅ Complete – rollout merged 2025-10-11 | 2025-10-20 | Maintain parity with OSV ecosystems; no action. |
|
||||
| Osv | BE-Conn-OSV | ✅ Complete – normalized rules shipping | 2025-10-20 | Watch for new ecosystems; refresh fixtures as needed. |
|
||||
| Ics.Cisa | BE-Conn-ICS-CISA | ⚠️ Decision pending – exact SemVer promotion due 2025-10-23 | 2025-10-20 | Promote primitives or request new scheme; `src/StellaOps.Concelier.Connector.Ics.Cisa/TASKS.md`. |
|
||||
| Kisa | BE-Conn-KISA | ⚠️ Proposal drafting – firmware scheme due 2025-10-24 | 2025-10-20 | Finalise `kisa.build` proposal with Models; update mapper/tests; `src/StellaOps.Concelier.Connector.Kisa/TASKS.md`. |
|
||||
| Ics.Cisa | BE-Conn-ICS-CISA | ⚠️ Decision pending – exact SemVer promotion due 2025-10-23 | 2025-10-20 | Promote primitives or request new scheme; `src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Ics.Cisa/TASKS.md`. |
|
||||
| Kisa | BE-Conn-KISA | ⚠️ Proposal drafting – firmware scheme due 2025-10-24 | 2025-10-20 | Finalise `kisa.build` proposal with Models; update mapper/tests; `src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Kisa/TASKS.md`. |
|
||||
| Ru.Bdu | BE-Conn-BDU | ✅ Complete – `ru-bdu.raw` rules live | 2025-10-20 | Continue monitoring UTF-8 handling; no action. |
|
||||
| Ru.Nkcki | BE-Conn-Nkcki | ✅ Complete – normalized rules emitted | 2025-10-20 | Maintain transliteration guidance; no action. |
|
||||
| Vndr.Apple | BE-Conn-Apple | ✅ Complete – normalized arrays emitting | 2025-10-20 | Add beta-channel coverage follow-up; see module README. |
|
||||
| Vndr.Cisco | BE-Conn-Cisco | ⚠️ DOING – normalized promotion branch open (due 2025-10-21) | 2025-10-20 | Merge helper branch, refresh fixtures, post counters; `src/StellaOps.Concelier.Connector.Vndr.Cisco/TASKS.md`. |
|
||||
| Vndr.Cisco | BE-Conn-Cisco | ⚠️ DOING – normalized promotion branch open (due 2025-10-21) | 2025-10-20 | Merge helper branch, refresh fixtures, post counters; `src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Vndr.Cisco/TASKS.md`. |
|
||||
| Vndr.Msrc | BE-Conn-MSRC | ✅ Complete – `msrc.build` rules emitting | 2025-10-20 | Monitor monthly rollups; no action. |
|
||||
| Nvd | BE-Conn-NVD | ✅ Complete – normalized SemVer output live | 2025-10-20 | Keep provenance aligned with CVE IDs; monitor export parity toggle. |
|
||||
|
||||
|
||||
@@ -1,27 +1,27 @@
|
||||
# Policy Schema Export Automation
|
||||
|
||||
This utility generates JSON Schema documents for the Policy Engine run contracts.
|
||||
|
||||
## Command
|
||||
|
||||
```
|
||||
scripts/export-policy-schemas.sh [output-directory]
|
||||
```
|
||||
|
||||
When no output directory is supplied, schemas are written to `docs/schemas/`.
|
||||
|
||||
The exporter builds against `StellaOps.Scheduler.Models` and emits:
|
||||
|
||||
- `policy-run-request.schema.json`
|
||||
- `policy-run-status.schema.json`
|
||||
- `policy-diff-summary.schema.json`
|
||||
- `policy-explain-trace.schema.json`
|
||||
|
||||
The build pipeline (`.gitea/workflows/build-test-deploy.yml`, job **Export policy run schemas**) runs this script on every push and pull request. Exports land under `artifacts/policy-schemas/<commit>/`, are published as the `policy-schema-exports` artifact, and changes trigger a Slack post to `#policy-engine` via the `POLICY_ENGINE_SCHEMA_WEBHOOK` secret. A unified diff is stored alongside the exports for downstream consumers.
|
||||
|
||||
## CI integration checklist
|
||||
|
||||
- [x] Invoke the script in the DevOps pipeline (see `DEVOPS-POLICY-20-004`).
|
||||
- [x] Publish the generated schemas as pipeline artifacts.
|
||||
- [x] Notify downstream consumers when schemas change (Slack `#policy-engine`, changelog snippet).
|
||||
- [ ] Gate CLI validation once schema artifacts are available.
|
||||
# Policy Schema Export Automation
|
||||
|
||||
This utility generates JSON Schema documents for the Policy Engine run contracts.
|
||||
|
||||
## Command
|
||||
|
||||
```
|
||||
scripts/export-policy-schemas.sh [output-directory]
|
||||
```
|
||||
|
||||
When no output directory is supplied, schemas are written to `docs/schemas/`.
|
||||
|
||||
The exporter builds against `StellaOps.Scheduler.Models` and emits:
|
||||
|
||||
- `policy-run-request.schema.json`
|
||||
- `policy-run-status.schema.json`
|
||||
- `policy-diff-summary.schema.json`
|
||||
- `policy-explain-trace.schema.json`
|
||||
|
||||
The build pipeline (`.gitea/workflows/build-test-deploy.yml`, job **Export policy run schemas**) runs this script on every push and pull request. Exports land under `artifacts/policy-schemas/<commit>/`, are published as the `policy-schema-exports` artifact, and changes trigger a Slack post to `#policy-engine` via the `POLICY_ENGINE_SCHEMA_WEBHOOK` secret. A unified diff is stored alongside the exports for downstream consumers.
|
||||
|
||||
## CI integration checklist
|
||||
|
||||
- [x] Invoke the script in the DevOps pipeline (see `DEVOPS-POLICY-20-004`).
|
||||
- [x] Publish the generated schemas as pipeline artifacts.
|
||||
- [x] Notify downstream consumers when schemas change (Slack `#policy-engine`, changelog snippet).
|
||||
- [ ] Gate CLI validation once schema artifacts are available.
|
||||
|
||||
@@ -1,121 +1,121 @@
|
||||
# Scanner Orchestrator Events (ORCH-SVC-38-101)
|
||||
|
||||
Last updated: 2025-10-26
|
||||
|
||||
The Notifications Studio initiative (NOTIFY-SVC-38-001) and orchestrator backlog (ORCH-SVC-38-101) standardise how platform services emit lifecycle events. This document describes the Scanner WebService contract for the new **orchestrator envelopes** (`scanner.event.*`) and how they supersede the legacy Redis-backed `scanner.report.ready` / `scanner.scan.completed` events.
|
||||
|
||||
## 1. Envelope overview
|
||||
|
||||
Orchestrator events share a deterministic JSON envelope:
|
||||
|
||||
| Field | Type | Notes |
|
||||
|-------|------|-------|
|
||||
| `eventId` | `uuid` | Globally unique identifier generated per occurrence. |
|
||||
| `kind` | `string` | Event identifier; Scanner emits `scanner.event.report.ready` and `scanner.event.scan.completed`. |
|
||||
| `version` | `integer` | Schema version. Initial release uses `1`. |
|
||||
| `tenant` | `string` | Tenant that owns the scan/report. Mirrors Authority claims. |
|
||||
| `occurredAt` | `date-time` | UTC instant when the underlying state transition happened (e.g., report persisted). |
|
||||
| `recordedAt` | `date-time` | UTC instant when the event was durably written. Optional but recommended. |
|
||||
| `source` | `string` | Producer identifier (`scanner.webservice`). |
|
||||
| `idempotencyKey` | `string` | Deterministic key for duplicate suppression (see §4). |
|
||||
| `correlationId` | `string` | Maps back to the API request or scan identifier. |
|
||||
| `traceId` / `spanId` | `string` | W3C trace context propagated into downstream telemetry. |
|
||||
| `scope` | `object` | Describes the affected artefact. Requires `repo` and `digest`; optional `namespace`, `component`, `image`. |
|
||||
| `attributes` | `object` | Flat string map for frequently queried metadata (e.g., policy revision). |
|
||||
| `payload` | `object` | Event-specific body (see §2). |
|
||||
|
||||
Canonical schemas live under `docs/events/scanner.event.*@1.json`. Samples that round-trip through `NotifyCanonicalJsonSerializer` are stored in `docs/events/samples/`.
|
||||
|
||||
## 2. Event kinds and payloads
|
||||
|
||||
### 2.1 `scanner.event.report.ready`
|
||||
|
||||
Emitted once a signed report is persisted and attested. Payload highlights:
|
||||
|
||||
- `reportId` / `scanId` — identifiers for the persisted report and originating scan. Until Scan IDs are surfaced by the API, `scanId` mirrors `reportId` so downstream correlators can stabilise on a single key.
|
||||
- **Attributes:** `reportId`, `policyRevisionId`, `policyDigest`, `verdict` — pre-sorted for deterministic routing.
|
||||
- **Links:**
|
||||
- `ui` → `/ui/reports/{reportId}` on the current host.
|
||||
- `report` → `{apiBasePath}/{reportsSegment}/{reportId}` (defaults to `/api/v1/reports/{reportId}`).
|
||||
- `policy` → `{apiBasePath}/{policySegment}/revisions/{revisionId}` when a revision is present.
|
||||
- `attestation` → `/ui/attestations/{reportId}` when a DSSE envelope is included.
|
||||
- `imageDigest` — OCI image digest associated with the analysis.
|
||||
- `generatedAt` — report generation timestamp (ISO-8601 UTC).
|
||||
- `verdict` — `pass`, `warn`, or `fail` after policy evaluation.
|
||||
- `summary` — blocked/warned/ignored/quieted counters (all non-negative integers).
|
||||
- `delta` — newly critical/high counts and optional `kev` array.
|
||||
- `quietedFindingCount` — mirrors `summary.quieted`.
|
||||
- `policy` — revision metadata (`digest`, `revisionId`) surfaced for routing.
|
||||
- `links` — UI/report/policy URLs suitable for operators.
|
||||
- `dsse` — embedded DSSE envelope (payload, type, signature list).
|
||||
- `report` — canonical report document; identical to the DSSE payload.
|
||||
|
||||
Schema: `docs/events/scanner.event.report.ready@1.json`
|
||||
Sample: `docs/events/samples/scanner.event.report.ready@1.sample.json`
|
||||
|
||||
### 2.2 `scanner.event.scan.completed`
|
||||
|
||||
Emitted after scan execution finishes (success or policy failure). Payload highlights:
|
||||
|
||||
- `reportId` / `scanId` / `imageDigest` — identifiers mirroring the report-ready event. As with the report-ready payload, `scanId` currently mirrors `reportId` as a temporary shim.
|
||||
- **Attributes:** `reportId`, `policyRevisionId`, `policyDigest`, `verdict`.
|
||||
- **Links:** same as above (`ui`, `report`, `policy`) with `attestation` populated when DSSE metadata exists.
|
||||
- `verdict`, `summary`, `delta`, `policy` — same semantics as above.
|
||||
- `findings` — array of surfaced findings with `id`, `severity`, optional `cve`, `purl`, and `reachability`.
|
||||
- `links`, `dsse`, `report` — same structure as §2.1 (allows Notifier to reuse signatures).
|
||||
|
||||
Schema: `docs/events/scanner.event.scan.completed@1.json`
|
||||
Sample: `docs/events/samples/scanner.event.scan.completed@1.sample.json`
|
||||
|
||||
### 2.3 Relationship to legacy events
|
||||
|
||||
| Legacy Redis event | Replacement orchestrator event | Notes |
|
||||
|--------------------|-------------------------------|-------|
|
||||
| `scanner.report.ready` | `scanner.event.report.ready` | Adds versioning, idempotency, trace context. Payload is a superset of the legacy fields. |
|
||||
| `scanner.scan.completed` | `scanner.event.scan.completed` | Same data plus explicit scan identifiers and orchestrator metadata. |
|
||||
|
||||
Legacy schemas remain for backwards-compatibility during migration, but new integrations **must** target the orchestrator variants.
|
||||
|
||||
## 3. Deterministic serialization
|
||||
|
||||
- Producers must serialise events using `NotifyCanonicalJsonSerializer` to guarantee consistent key ordering and whitespace.
|
||||
- Timestamps (`occurredAt`, `recordedAt`, `payload.generatedAt`) use `DateTimeOffset.UtcDateTime.ToString("O")`.
|
||||
- Payload arrays (`delta.kev`, `findings`) should be pre-sorted (e.g., alphabetical CVE order) so hash-based consumers remain stable.
|
||||
- Optional fields are omitted rather than emitted as `null`.
|
||||
|
||||
## 4. Idempotency and correlation
|
||||
|
||||
Idempotency keys dedupe repeated publishes and align with the orchestrator’s outbox pattern:
|
||||
|
||||
| Event kind | Idempotency key template |
|
||||
|------------|-------------------------|
|
||||
| `scanner.event.report.ready` | `scanner.event.report.ready:<tenant>:<reportId>` |
|
||||
| `scanner.event.scan.completed` | `scanner.event.scan.completed:<tenant>:<scanId>` |
|
||||
|
||||
Keys are ASCII lowercase; components should be trimmed and validated before concatenation. Retries must reuse the same key.
|
||||
|
||||
`correlationId` should match the scan identifier that appears in REST responses (`scanId`). Re-using the same value across the pair of events allows Notifier and orchestrator analytics to stitch lifecycle data together.
|
||||
|
||||
## 5. Versioning and evolution
|
||||
|
||||
- Increment the `version` field and the `@<version>` suffix for **breaking** changes (field removals, type changes, semantic shifts).
|
||||
- Additive optional fields may remain within version 1; update the JSON schema and samples accordingly.
|
||||
- When introducing `@2`, keep the `@1` schema/docs in place until orchestrator subscribers confirm migration.
|
||||
|
||||
## 6. Consumer checklist
|
||||
|
||||
1. Validate incoming payloads against the schema for the targeted version.
|
||||
2. Use `idempotencyKey` for dedupe, not `eventId`.
|
||||
3. Map `traceId`/`spanId` into telemetry spans to preserve causality.
|
||||
4. Prefer `payload.report` → `policy.revisionId` when populating templates; the top-level `attributes` are convenience duplicates for quick routing.
|
||||
5. Reserve the legacy Redis events for transitional compatibility only; downstream systems should subscribe to the orchestrator bus exposed by ORCH-SVC-38-101.
|
||||
|
||||
## 7. Implementation status and next actions
|
||||
|
||||
- **Scanner WebService** — `SCANNER-EVENTS-16-301` (blocked) and `SCANNER-EVENTS-16-302` (doing) track the production of these envelopes. The remaining blocker is the .NET 10 preview OpenAPI/Auth dependency drift that currently breaks `dotnet test`. Once Gateway and Notifier owners land the replacement packages, rerun the full test suite and capture fresh fixtures under `docs/events/samples/`.
|
||||
- **Gateway/Notifier consumers** — subscribe to the orchestrator stream documented in ORCH-SVC-38-101. When the Scanner tasks unblock, regenerate notifier contract tests against the sample events included here.
|
||||
- **Docs cadence** — update this file and the matching JSON schemas whenever payload fields change. Use the rehearsal checklist in `docs/ops/launch-cutover.md` to confirm downstream validation before the production cutover. Record gaps or newly required fields in `docs/ops/launch-readiness.md` so they land in the launch checklist.
|
||||
|
||||
---
|
||||
|
||||
**Imposed rule reminder:** work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.
|
||||
# Scanner Orchestrator Events (ORCH-SVC-38-101)
|
||||
|
||||
Last updated: 2025-10-26
|
||||
|
||||
The Notifications Studio initiative (NOTIFY-SVC-38-001) and orchestrator backlog (ORCH-SVC-38-101) standardise how platform services emit lifecycle events. This document describes the Scanner WebService contract for the new **orchestrator envelopes** (`scanner.event.*`) and how they supersede the legacy Redis-backed `scanner.report.ready` / `scanner.scan.completed` events.
|
||||
|
||||
## 1. Envelope overview
|
||||
|
||||
Orchestrator events share a deterministic JSON envelope:
|
||||
|
||||
| Field | Type | Notes |
|
||||
|-------|------|-------|
|
||||
| `eventId` | `uuid` | Globally unique identifier generated per occurrence. |
|
||||
| `kind` | `string` | Event identifier; Scanner emits `scanner.event.report.ready` and `scanner.event.scan.completed`. |
|
||||
| `version` | `integer` | Schema version. Initial release uses `1`. |
|
||||
| `tenant` | `string` | Tenant that owns the scan/report. Mirrors Authority claims. |
|
||||
| `occurredAt` | `date-time` | UTC instant when the underlying state transition happened (e.g., report persisted). |
|
||||
| `recordedAt` | `date-time` | UTC instant when the event was durably written. Optional but recommended. |
|
||||
| `source` | `string` | Producer identifier (`scanner.webservice`). |
|
||||
| `idempotencyKey` | `string` | Deterministic key for duplicate suppression (see §4). |
|
||||
| `correlationId` | `string` | Maps back to the API request or scan identifier. |
|
||||
| `traceId` / `spanId` | `string` | W3C trace context propagated into downstream telemetry. |
|
||||
| `scope` | `object` | Describes the affected artefact. Requires `repo` and `digest`; optional `namespace`, `component`, `image`. |
|
||||
| `attributes` | `object` | Flat string map for frequently queried metadata (e.g., policy revision). |
|
||||
| `payload` | `object` | Event-specific body (see §2). |
|
||||
|
||||
Canonical schemas live under `docs/events/scanner.event.*@1.json`. Samples that round-trip through `NotifyCanonicalJsonSerializer` are stored in `docs/events/samples/`.
|
||||
|
||||
## 2. Event kinds and payloads
|
||||
|
||||
### 2.1 `scanner.event.report.ready`
|
||||
|
||||
Emitted once a signed report is persisted and attested. Payload highlights:
|
||||
|
||||
- `reportId` / `scanId` — identifiers for the persisted report and originating scan. Until Scan IDs are surfaced by the API, `scanId` mirrors `reportId` so downstream correlators can stabilise on a single key.
|
||||
- **Attributes:** `reportId`, `policyRevisionId`, `policyDigest`, `verdict` — pre-sorted for deterministic routing.
|
||||
- **Links:**
|
||||
- `ui` → `/ui/reports/{reportId}` on the current host.
|
||||
- `report` → `{apiBasePath}/{reportsSegment}/{reportId}` (defaults to `/api/v1/reports/{reportId}`).
|
||||
- `policy` → `{apiBasePath}/{policySegment}/revisions/{revisionId}` when a revision is present.
|
||||
- `attestation` → `/ui/attestations/{reportId}` when a DSSE envelope is included.
|
||||
- `imageDigest` — OCI image digest associated with the analysis.
|
||||
- `generatedAt` — report generation timestamp (ISO-8601 UTC).
|
||||
- `verdict` — `pass`, `warn`, or `fail` after policy evaluation.
|
||||
- `summary` — blocked/warned/ignored/quieted counters (all non-negative integers).
|
||||
- `delta` — newly critical/high counts and optional `kev` array.
|
||||
- `quietedFindingCount` — mirrors `summary.quieted`.
|
||||
- `policy` — revision metadata (`digest`, `revisionId`) surfaced for routing.
|
||||
- `links` — UI/report/policy URLs suitable for operators.
|
||||
- `dsse` — embedded DSSE envelope (payload, type, signature list).
|
||||
- `report` — canonical report document; identical to the DSSE payload.
|
||||
|
||||
Schema: `docs/events/scanner.event.report.ready@1.json`
|
||||
Sample: `docs/events/samples/scanner.event.report.ready@1.sample.json`
|
||||
|
||||
### 2.2 `scanner.event.scan.completed`
|
||||
|
||||
Emitted after scan execution finishes (success or policy failure). Payload highlights:
|
||||
|
||||
- `reportId` / `scanId` / `imageDigest` — identifiers mirroring the report-ready event. As with the report-ready payload, `scanId` currently mirrors `reportId` as a temporary shim.
|
||||
- **Attributes:** `reportId`, `policyRevisionId`, `policyDigest`, `verdict`.
|
||||
- **Links:** same as above (`ui`, `report`, `policy`) with `attestation` populated when DSSE metadata exists.
|
||||
- `verdict`, `summary`, `delta`, `policy` — same semantics as above.
|
||||
- `findings` — array of surfaced findings with `id`, `severity`, optional `cve`, `purl`, and `reachability`.
|
||||
- `links`, `dsse`, `report` — same structure as §2.1 (allows Notifier to reuse signatures).
|
||||
|
||||
Schema: `docs/events/scanner.event.scan.completed@1.json`
|
||||
Sample: `docs/events/samples/scanner.event.scan.completed@1.sample.json`
|
||||
|
||||
### 2.3 Relationship to legacy events
|
||||
|
||||
| Legacy Redis event | Replacement orchestrator event | Notes |
|
||||
|--------------------|-------------------------------|-------|
|
||||
| `scanner.report.ready` | `scanner.event.report.ready` | Adds versioning, idempotency, trace context. Payload is a superset of the legacy fields. |
|
||||
| `scanner.scan.completed` | `scanner.event.scan.completed` | Same data plus explicit scan identifiers and orchestrator metadata. |
|
||||
|
||||
Legacy schemas remain for backwards-compatibility during migration, but new integrations **must** target the orchestrator variants.
|
||||
|
||||
## 3. Deterministic serialization
|
||||
|
||||
- Producers must serialise events using `NotifyCanonicalJsonSerializer` to guarantee consistent key ordering and whitespace.
|
||||
- Timestamps (`occurredAt`, `recordedAt`, `payload.generatedAt`) use `DateTimeOffset.UtcDateTime.ToString("O")`.
|
||||
- Payload arrays (`delta.kev`, `findings`) should be pre-sorted (e.g., alphabetical CVE order) so hash-based consumers remain stable.
|
||||
- Optional fields are omitted rather than emitted as `null`.
|
||||
|
||||
## 4. Idempotency and correlation
|
||||
|
||||
Idempotency keys dedupe repeated publishes and align with the orchestrator’s outbox pattern:
|
||||
|
||||
| Event kind | Idempotency key template |
|
||||
|------------|-------------------------|
|
||||
| `scanner.event.report.ready` | `scanner.event.report.ready:<tenant>:<reportId>` |
|
||||
| `scanner.event.scan.completed` | `scanner.event.scan.completed:<tenant>:<scanId>` |
|
||||
|
||||
Keys are ASCII lowercase; components should be trimmed and validated before concatenation. Retries must reuse the same key.
|
||||
|
||||
`correlationId` should match the scan identifier that appears in REST responses (`scanId`). Re-using the same value across the pair of events allows Notifier and orchestrator analytics to stitch lifecycle data together.
|
||||
|
||||
## 5. Versioning and evolution
|
||||
|
||||
- Increment the `version` field and the `@<version>` suffix for **breaking** changes (field removals, type changes, semantic shifts).
|
||||
- Additive optional fields may remain within version 1; update the JSON schema and samples accordingly.
|
||||
- When introducing `@2`, keep the `@1` schema/docs in place until orchestrator subscribers confirm migration.
|
||||
|
||||
## 6. Consumer checklist
|
||||
|
||||
1. Validate incoming payloads against the schema for the targeted version.
|
||||
2. Use `idempotencyKey` for dedupe, not `eventId`.
|
||||
3. Map `traceId`/`spanId` into telemetry spans to preserve causality.
|
||||
4. Prefer `payload.report` → `policy.revisionId` when populating templates; the top-level `attributes` are convenience duplicates for quick routing.
|
||||
5. Reserve the legacy Redis events for transitional compatibility only; downstream systems should subscribe to the orchestrator bus exposed by ORCH-SVC-38-101.
|
||||
|
||||
## 7. Implementation status and next actions
|
||||
|
||||
- **Scanner WebService** — `SCANNER-EVENTS-16-301` (blocked) and `SCANNER-EVENTS-16-302` (doing) track the production of these envelopes. The remaining blocker is the .NET 10 preview OpenAPI/Auth dependency drift that currently breaks `dotnet test`. Once Gateway and Notifier owners land the replacement packages, rerun the full test suite and capture fresh fixtures under `docs/events/samples/`.
|
||||
- **Gateway/Notifier consumers** — subscribe to the orchestrator stream documented in ORCH-SVC-38-101. When the Scanner tasks unblock, regenerate notifier contract tests against the sample events included here.
|
||||
- **Docs cadence** — update this file and the matching JSON schemas whenever payload fields change. Use the rehearsal checklist in `docs/ops/launch-cutover.md` to confirm downstream validation before the production cutover. Record gaps or newly required fields in `docs/ops/launch-readiness.md` so they land in the launch checklist.
|
||||
|
||||
---
|
||||
|
||||
**Imposed rule reminder:** work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.
|
||||
|
||||
@@ -1,93 +1,93 @@
|
||||
{
|
||||
"eventId": "6d2d1b77-f3c3-4f70-8a9d-6f2d0c8801ab",
|
||||
"kind": "scanner.event.report.ready",
|
||||
"version": 1,
|
||||
"tenant": "tenant-alpha",
|
||||
"occurredAt": "2025-10-19T12:34:56Z",
|
||||
"recordedAt": "2025-10-19T12:34:57Z",
|
||||
"source": "scanner.webservice",
|
||||
"idempotencyKey": "scanner.event.report.ready:tenant-alpha:report-abc",
|
||||
"correlationId": "report-abc",
|
||||
"traceId": "0af7651916cd43dd8448eb211c80319c",
|
||||
"spanId": "b7ad6b7169203331",
|
||||
"scope": {
|
||||
"namespace": "acme/edge",
|
||||
"repo": "api",
|
||||
"digest": "sha256:feedface"
|
||||
},
|
||||
"attributes": {
|
||||
"reportId": "report-abc",
|
||||
"policyRevisionId": "rev-42",
|
||||
"policyDigest": "digest-123",
|
||||
"verdict": "blocked"
|
||||
},
|
||||
"payload": {
|
||||
"reportId": "report-abc",
|
||||
"scanId": "report-abc",
|
||||
"imageDigest": "sha256:feedface",
|
||||
"generatedAt": "2025-10-19T12:34:56Z",
|
||||
"verdict": "fail",
|
||||
"summary": {
|
||||
"total": 1,
|
||||
"blocked": 1,
|
||||
"warned": 0,
|
||||
"ignored": 0,
|
||||
"quieted": 0
|
||||
},
|
||||
"delta": {
|
||||
"newCritical": 1,
|
||||
"kev": [
|
||||
"CVE-2024-9999"
|
||||
]
|
||||
},
|
||||
"quietedFindingCount": 0,
|
||||
"policy": {
|
||||
"digest": "digest-123",
|
||||
"revisionId": "rev-42"
|
||||
},
|
||||
"links": {
|
||||
"ui": "https://scanner.example/ui/reports/report-abc",
|
||||
"report": "https://scanner.example/api/v1/reports/report-abc",
|
||||
"policy": "https://scanner.example/api/v1/policy/revisions/rev-42",
|
||||
"attestation": "https://scanner.example/ui/attestations/report-abc"
|
||||
},
|
||||
"dsse": {
|
||||
"payloadType": "application/vnd.stellaops.report+json",
|
||||
"payload": "eyJyZXBvcnRJZCI6InJlcG9ydC1hYmMiLCJpbWFnZURpZ2VzdCI6InNoYTI1NjpmZWVkZmFjZSIsImdlbmVyYXRlZEF0IjoiMjAyNS0xMC0xOVQxMjozNDo1NiswMDowMCIsInZlcmRpY3QiOiJibG9ja2VkIiwicG9saWN5Ijp7InJldmlzaW9uSWQiOiJyZXYtNDIiLCJkaWdlc3QiOiJkaWdlc3QtMTIzIn0sInN1bW1hcnkiOnsidG90YWwiOjEsImJsb2NrZWQiOjEsIndhcm5lZCI6MCwiaWdub3JlZCI6MCwicXVpZXRlZCI6MH0sInZlcmRpY3RzIjpbeyJmaW5kaW5nSWQiOiJmaW5kaW5nLTEiLCJzdGF0dXMiOiJCbG9ja2VkIiwic2NvcmUiOjQ3LjUsInNvdXJjZVRydXN0IjoiTlZEIiwicmVhY2hhYmlsaXR5IjoicnVudGltZSJ9XSwiaXNzdWVzIjpbXX0=",
|
||||
"signatures": [
|
||||
{
|
||||
"keyId": "test-key",
|
||||
"algorithm": "hs256",
|
||||
"signature": "signature-value"
|
||||
}
|
||||
]
|
||||
},
|
||||
"report": {
|
||||
"reportId": "report-abc",
|
||||
"generatedAt": "2025-10-19T12:34:56Z",
|
||||
"imageDigest": "sha256:feedface",
|
||||
"policy": {
|
||||
"digest": "digest-123",
|
||||
"revisionId": "rev-42"
|
||||
},
|
||||
"summary": {
|
||||
"total": 1,
|
||||
"blocked": 1,
|
||||
"warned": 0,
|
||||
"ignored": 0,
|
||||
"quieted": 0
|
||||
},
|
||||
"verdict": "blocked",
|
||||
"verdicts": [
|
||||
{
|
||||
"findingId": "finding-1",
|
||||
"status": "Blocked",
|
||||
"score": 47.5,
|
||||
"sourceTrust": "NVD",
|
||||
"reachability": "runtime"
|
||||
}
|
||||
],
|
||||
"issues": []
|
||||
}
|
||||
}
|
||||
}
|
||||
{
|
||||
"eventId": "6d2d1b77-f3c3-4f70-8a9d-6f2d0c8801ab",
|
||||
"kind": "scanner.event.report.ready",
|
||||
"version": 1,
|
||||
"tenant": "tenant-alpha",
|
||||
"occurredAt": "2025-10-19T12:34:56Z",
|
||||
"recordedAt": "2025-10-19T12:34:57Z",
|
||||
"source": "scanner.webservice",
|
||||
"idempotencyKey": "scanner.event.report.ready:tenant-alpha:report-abc",
|
||||
"correlationId": "report-abc",
|
||||
"traceId": "0af7651916cd43dd8448eb211c80319c",
|
||||
"spanId": "b7ad6b7169203331",
|
||||
"scope": {
|
||||
"namespace": "acme/edge",
|
||||
"repo": "api",
|
||||
"digest": "sha256:feedface"
|
||||
},
|
||||
"attributes": {
|
||||
"reportId": "report-abc",
|
||||
"policyRevisionId": "rev-42",
|
||||
"policyDigest": "digest-123",
|
||||
"verdict": "blocked"
|
||||
},
|
||||
"payload": {
|
||||
"reportId": "report-abc",
|
||||
"scanId": "report-abc",
|
||||
"imageDigest": "sha256:feedface",
|
||||
"generatedAt": "2025-10-19T12:34:56Z",
|
||||
"verdict": "fail",
|
||||
"summary": {
|
||||
"total": 1,
|
||||
"blocked": 1,
|
||||
"warned": 0,
|
||||
"ignored": 0,
|
||||
"quieted": 0
|
||||
},
|
||||
"delta": {
|
||||
"newCritical": 1,
|
||||
"kev": [
|
||||
"CVE-2024-9999"
|
||||
]
|
||||
},
|
||||
"quietedFindingCount": 0,
|
||||
"policy": {
|
||||
"digest": "digest-123",
|
||||
"revisionId": "rev-42"
|
||||
},
|
||||
"links": {
|
||||
"ui": "https://scanner.example/ui/reports/report-abc",
|
||||
"report": "https://scanner.example/api/v1/reports/report-abc",
|
||||
"policy": "https://scanner.example/api/v1/policy/revisions/rev-42",
|
||||
"attestation": "https://scanner.example/ui/attestations/report-abc"
|
||||
},
|
||||
"dsse": {
|
||||
"payloadType": "application/vnd.stellaops.report+json",
|
||||
"payload": "eyJyZXBvcnRJZCI6InJlcG9ydC1hYmMiLCJpbWFnZURpZ2VzdCI6InNoYTI1NjpmZWVkZmFjZSIsImdlbmVyYXRlZEF0IjoiMjAyNS0xMC0xOVQxMjozNDo1NiswMDowMCIsInZlcmRpY3QiOiJibG9ja2VkIiwicG9saWN5Ijp7InJldmlzaW9uSWQiOiJyZXYtNDIiLCJkaWdlc3QiOiJkaWdlc3QtMTIzIn0sInN1bW1hcnkiOnsidG90YWwiOjEsImJsb2NrZWQiOjEsIndhcm5lZCI6MCwiaWdub3JlZCI6MCwicXVpZXRlZCI6MH0sInZlcmRpY3RzIjpbeyJmaW5kaW5nSWQiOiJmaW5kaW5nLTEiLCJzdGF0dXMiOiJCbG9ja2VkIiwic2NvcmUiOjQ3LjUsInNvdXJjZVRydXN0IjoiTlZEIiwicmVhY2hhYmlsaXR5IjoicnVudGltZSJ9XSwiaXNzdWVzIjpbXX0=",
|
||||
"signatures": [
|
||||
{
|
||||
"keyId": "test-key",
|
||||
"algorithm": "hs256",
|
||||
"signature": "signature-value"
|
||||
}
|
||||
]
|
||||
},
|
||||
"report": {
|
||||
"reportId": "report-abc",
|
||||
"generatedAt": "2025-10-19T12:34:56Z",
|
||||
"imageDigest": "sha256:feedface",
|
||||
"policy": {
|
||||
"digest": "digest-123",
|
||||
"revisionId": "rev-42"
|
||||
},
|
||||
"summary": {
|
||||
"total": 1,
|
||||
"blocked": 1,
|
||||
"warned": 0,
|
||||
"ignored": 0,
|
||||
"quieted": 0
|
||||
},
|
||||
"verdict": "blocked",
|
||||
"verdicts": [
|
||||
{
|
||||
"findingId": "finding-1",
|
||||
"status": "Blocked",
|
||||
"score": 47.5,
|
||||
"sourceTrust": "NVD",
|
||||
"reachability": "runtime"
|
||||
}
|
||||
],
|
||||
"issues": []
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1,99 +1,99 @@
|
||||
{
|
||||
"eventId": "08a6de24-4a94-4d14-8432-9d14f36f6da3",
|
||||
"kind": "scanner.event.scan.completed",
|
||||
"version": 1,
|
||||
"tenant": "tenant-alpha",
|
||||
"occurredAt": "2025-10-19T12:34:56Z",
|
||||
"recordedAt": "2025-10-19T12:34:57Z",
|
||||
"source": "scanner.webservice",
|
||||
"idempotencyKey": "scanner.event.scan.completed:tenant-alpha:report-abc",
|
||||
"correlationId": "report-abc",
|
||||
"traceId": "4bf92f3577b34da6a3ce929d0e0e4736",
|
||||
"scope": {
|
||||
"namespace": "acme/edge",
|
||||
"repo": "api",
|
||||
"digest": "sha256:feedface"
|
||||
},
|
||||
"attributes": {
|
||||
"reportId": "report-abc",
|
||||
"policyRevisionId": "rev-42",
|
||||
"policyDigest": "digest-123",
|
||||
"verdict": "blocked"
|
||||
},
|
||||
"payload": {
|
||||
"reportId": "report-abc",
|
||||
"scanId": "report-abc",
|
||||
"imageDigest": "sha256:feedface",
|
||||
"verdict": "fail",
|
||||
"summary": {
|
||||
"total": 1,
|
||||
"blocked": 1,
|
||||
"warned": 0,
|
||||
"ignored": 0,
|
||||
"quieted": 0
|
||||
},
|
||||
"delta": {
|
||||
"newCritical": 1,
|
||||
"kev": [
|
||||
"CVE-2024-9999"
|
||||
]
|
||||
},
|
||||
"policy": {
|
||||
"digest": "digest-123",
|
||||
"revisionId": "rev-42"
|
||||
},
|
||||
"findings": [
|
||||
{
|
||||
"id": "finding-1",
|
||||
"severity": "Critical",
|
||||
"cve": "CVE-2024-9999",
|
||||
"purl": "pkg:docker/acme/edge-api@sha256-feedface",
|
||||
"reachability": "runtime"
|
||||
}
|
||||
],
|
||||
"links": {
|
||||
"ui": "https://scanner.example/ui/reports/report-abc",
|
||||
"report": "https://scanner.example/api/v1/reports/report-abc",
|
||||
"policy": "https://scanner.example/api/v1/policy/revisions/rev-42",
|
||||
"attestation": "https://scanner.example/ui/attestations/report-abc"
|
||||
},
|
||||
"dsse": {
|
||||
"payloadType": "application/vnd.stellaops.report+json",
|
||||
"payload": "eyJyZXBvcnRJZCI6InJlcG9ydC1hYmMiLCJpbWFnZURpZ2VzdCI6InNoYTI1NjpmZWVkZmFjZSIsImdlbmVyYXRlZEF0IjoiMjAyNS0xMC0xOVQxMjozNDo1NiswMDowMCIsInZlcmRpY3QiOiJibG9ja2VkIiwicG9saWN5Ijp7InJldmlzaW9uSWQiOiJyZXYtNDIiLCJkaWdlc3QiOiJkaWdlc3QtMTIzIn0sInN1bW1hcnkiOnsidG90YWwiOjEsImJsb2NrZWQiOjEsIndhcm5lZCI6MCwiaWdub3JlZCI6MCwicXVpZXRlZCI6MH0sInZlcmRpY3RzIjpbeyJmaW5kaW5nSWQiOiJmaW5kaW5nLTEiLCJzdGF0dXMiOiJCbG9ja2VkIiwic2NvcmUiOjQ3LjUsInNvdXJjZVRydXN0IjoiTlZEIiwicmVhY2hhYmlsaXR5IjoicnVudGltZSJ9XSwiaXNzdWVzIjpbXX0=",
|
||||
"signatures": [
|
||||
{
|
||||
"keyId": "test-key",
|
||||
"algorithm": "hs256",
|
||||
"signature": "signature-value"
|
||||
}
|
||||
]
|
||||
},
|
||||
"report": {
|
||||
"reportId": "report-abc",
|
||||
"generatedAt": "2025-10-19T12:34:56Z",
|
||||
"imageDigest": "sha256:feedface",
|
||||
"policy": {
|
||||
"digest": "digest-123",
|
||||
"revisionId": "rev-42"
|
||||
},
|
||||
"summary": {
|
||||
"total": 1,
|
||||
"blocked": 1,
|
||||
"warned": 0,
|
||||
"ignored": 0,
|
||||
"quieted": 0
|
||||
},
|
||||
"verdict": "blocked",
|
||||
"verdicts": [
|
||||
{
|
||||
"findingId": "finding-1",
|
||||
"status": "Blocked",
|
||||
"score": 47.5,
|
||||
"sourceTrust": "NVD",
|
||||
"reachability": "runtime"
|
||||
}
|
||||
],
|
||||
"issues": []
|
||||
}
|
||||
}
|
||||
}
|
||||
{
|
||||
"eventId": "08a6de24-4a94-4d14-8432-9d14f36f6da3",
|
||||
"kind": "scanner.event.scan.completed",
|
||||
"version": 1,
|
||||
"tenant": "tenant-alpha",
|
||||
"occurredAt": "2025-10-19T12:34:56Z",
|
||||
"recordedAt": "2025-10-19T12:34:57Z",
|
||||
"source": "scanner.webservice",
|
||||
"idempotencyKey": "scanner.event.scan.completed:tenant-alpha:report-abc",
|
||||
"correlationId": "report-abc",
|
||||
"traceId": "4bf92f3577b34da6a3ce929d0e0e4736",
|
||||
"scope": {
|
||||
"namespace": "acme/edge",
|
||||
"repo": "api",
|
||||
"digest": "sha256:feedface"
|
||||
},
|
||||
"attributes": {
|
||||
"reportId": "report-abc",
|
||||
"policyRevisionId": "rev-42",
|
||||
"policyDigest": "digest-123",
|
||||
"verdict": "blocked"
|
||||
},
|
||||
"payload": {
|
||||
"reportId": "report-abc",
|
||||
"scanId": "report-abc",
|
||||
"imageDigest": "sha256:feedface",
|
||||
"verdict": "fail",
|
||||
"summary": {
|
||||
"total": 1,
|
||||
"blocked": 1,
|
||||
"warned": 0,
|
||||
"ignored": 0,
|
||||
"quieted": 0
|
||||
},
|
||||
"delta": {
|
||||
"newCritical": 1,
|
||||
"kev": [
|
||||
"CVE-2024-9999"
|
||||
]
|
||||
},
|
||||
"policy": {
|
||||
"digest": "digest-123",
|
||||
"revisionId": "rev-42"
|
||||
},
|
||||
"findings": [
|
||||
{
|
||||
"id": "finding-1",
|
||||
"severity": "Critical",
|
||||
"cve": "CVE-2024-9999",
|
||||
"purl": "pkg:docker/acme/edge-api@sha256-feedface",
|
||||
"reachability": "runtime"
|
||||
}
|
||||
],
|
||||
"links": {
|
||||
"ui": "https://scanner.example/ui/reports/report-abc",
|
||||
"report": "https://scanner.example/api/v1/reports/report-abc",
|
||||
"policy": "https://scanner.example/api/v1/policy/revisions/rev-42",
|
||||
"attestation": "https://scanner.example/ui/attestations/report-abc"
|
||||
},
|
||||
"dsse": {
|
||||
"payloadType": "application/vnd.stellaops.report+json",
|
||||
"payload": "eyJyZXBvcnRJZCI6InJlcG9ydC1hYmMiLCJpbWFnZURpZ2VzdCI6InNoYTI1NjpmZWVkZmFjZSIsImdlbmVyYXRlZEF0IjoiMjAyNS0xMC0xOVQxMjozNDo1NiswMDowMCIsInZlcmRpY3QiOiJibG9ja2VkIiwicG9saWN5Ijp7InJldmlzaW9uSWQiOiJyZXYtNDIiLCJkaWdlc3QiOiJkaWdlc3QtMTIzIn0sInN1bW1hcnkiOnsidG90YWwiOjEsImJsb2NrZWQiOjEsIndhcm5lZCI6MCwiaWdub3JlZCI6MCwicXVpZXRlZCI6MH0sInZlcmRpY3RzIjpbeyJmaW5kaW5nSWQiOiJmaW5kaW5nLTEiLCJzdGF0dXMiOiJCbG9ja2VkIiwic2NvcmUiOjQ3LjUsInNvdXJjZVRydXN0IjoiTlZEIiwicmVhY2hhYmlsaXR5IjoicnVudGltZSJ9XSwiaXNzdWVzIjpbXX0=",
|
||||
"signatures": [
|
||||
{
|
||||
"keyId": "test-key",
|
||||
"algorithm": "hs256",
|
||||
"signature": "signature-value"
|
||||
}
|
||||
]
|
||||
},
|
||||
"report": {
|
||||
"reportId": "report-abc",
|
||||
"generatedAt": "2025-10-19T12:34:56Z",
|
||||
"imageDigest": "sha256:feedface",
|
||||
"policy": {
|
||||
"digest": "digest-123",
|
||||
"revisionId": "rev-42"
|
||||
},
|
||||
"summary": {
|
||||
"total": 1,
|
||||
"blocked": 1,
|
||||
"warned": 0,
|
||||
"ignored": 0,
|
||||
"quieted": 0
|
||||
},
|
||||
"verdict": "blocked",
|
||||
"verdicts": [
|
||||
{
|
||||
"findingId": "finding-1",
|
||||
"status": "Blocked",
|
||||
"score": 47.5,
|
||||
"sourceTrust": "NVD",
|
||||
"reachability": "runtime"
|
||||
}
|
||||
],
|
||||
"issues": []
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1,36 +1,36 @@
|
||||
{
|
||||
"eventId": "4d33c19c-1c8a-44d1-9954-1d5e98b2af71",
|
||||
"kind": "scheduler.graph.job.completed",
|
||||
"tenant": "tenant-alpha",
|
||||
"ts": "2025-10-26T12:00:45Z",
|
||||
"payload": {
|
||||
"jobType": "build",
|
||||
"status": "completed",
|
||||
"occurredAt": "2025-10-26T12:00:45Z",
|
||||
"job": {
|
||||
"schemaVersion": "scheduler.graph-build-job@1",
|
||||
"id": "gbj_20251026a",
|
||||
"tenantId": "tenant-alpha",
|
||||
"sbomId": "sbom_20251026",
|
||||
"sbomVersionId": "sbom_ver_20251026",
|
||||
"sbomDigest": "sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef",
|
||||
"graphSnapshotId": "graph_snap_20251026",
|
||||
"status": "completed",
|
||||
"trigger": "sbom-version",
|
||||
"attempts": 1,
|
||||
"cartographerJobId": "carto_job_42",
|
||||
"correlationId": "evt_svc_987",
|
||||
"createdAt": "2025-10-26T12:00:00+00:00",
|
||||
"startedAt": "2025-10-26T12:00:05+00:00",
|
||||
"completedAt": "2025-10-26T12:00:45+00:00",
|
||||
"metadata": {
|
||||
"sbomEventId": "sbom_evt_20251026"
|
||||
}
|
||||
},
|
||||
"resultUri": "oras://cartographer/offline/tenant-alpha/graph_snap_20251026"
|
||||
},
|
||||
"attributes": {
|
||||
"cartographerCluster": "offline-kit",
|
||||
"plannerShard": "graph-builders-01"
|
||||
}
|
||||
}
|
||||
{
|
||||
"eventId": "4d33c19c-1c8a-44d1-9954-1d5e98b2af71",
|
||||
"kind": "scheduler.graph.job.completed",
|
||||
"tenant": "tenant-alpha",
|
||||
"ts": "2025-10-26T12:00:45Z",
|
||||
"payload": {
|
||||
"jobType": "build",
|
||||
"status": "completed",
|
||||
"occurredAt": "2025-10-26T12:00:45Z",
|
||||
"job": {
|
||||
"schemaVersion": "scheduler.graph-build-job@1",
|
||||
"id": "gbj_20251026a",
|
||||
"tenantId": "tenant-alpha",
|
||||
"sbomId": "sbom_20251026",
|
||||
"sbomVersionId": "sbom_ver_20251026",
|
||||
"sbomDigest": "sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef",
|
||||
"graphSnapshotId": "graph_snap_20251026",
|
||||
"status": "completed",
|
||||
"trigger": "sbom-version",
|
||||
"attempts": 1,
|
||||
"cartographerJobId": "carto_job_42",
|
||||
"correlationId": "evt_svc_987",
|
||||
"createdAt": "2025-10-26T12:00:00+00:00",
|
||||
"startedAt": "2025-10-26T12:00:05+00:00",
|
||||
"completedAt": "2025-10-26T12:00:45+00:00",
|
||||
"metadata": {
|
||||
"sbomEventId": "sbom_evt_20251026"
|
||||
}
|
||||
},
|
||||
"resultUri": "oras://cartographer/offline/tenant-alpha/graph_snap_20251026"
|
||||
},
|
||||
"attributes": {
|
||||
"cartographerCluster": "offline-kit",
|
||||
"plannerShard": "graph-builders-01"
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1,164 +1,164 @@
|
||||
{
|
||||
"$id": "https://stella-ops.org/schemas/events/scanner.event.report.ready@1.json",
|
||||
"$schema": "http://json-schema.org/draft-07/schema#",
|
||||
"title": "Scanner orchestrator event – report ready (v1)",
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": [
|
||||
"eventId",
|
||||
"kind",
|
||||
"version",
|
||||
"tenant",
|
||||
"occurredAt",
|
||||
"source",
|
||||
"idempotencyKey",
|
||||
"payload"
|
||||
],
|
||||
"properties": {
|
||||
"eventId": {
|
||||
"type": "string",
|
||||
"format": "uuid",
|
||||
"description": "Globally unique identifier for this occurrence."
|
||||
},
|
||||
"kind": {
|
||||
"const": "scanner.event.report.ready",
|
||||
"description": "Event kind identifier consumed by orchestrator subscribers."
|
||||
},
|
||||
"version": {
|
||||
"const": 1,
|
||||
"description": "Schema version for orchestrator envelopes."
|
||||
},
|
||||
"tenant": {
|
||||
"type": "string",
|
||||
"description": "Tenant that owns the scan/report."
|
||||
},
|
||||
"occurredAt": {
|
||||
"type": "string",
|
||||
"format": "date-time",
|
||||
"description": "Timestamp (UTC) when the report transitioned to ready."
|
||||
},
|
||||
"recordedAt": {
|
||||
"type": "string",
|
||||
"format": "date-time",
|
||||
"description": "Timestamp (UTC) when the event was persisted. Optional."
|
||||
},
|
||||
"source": {
|
||||
"type": "string",
|
||||
"description": "Producer identifier, e.g. `scanner.webservice`."
|
||||
},
|
||||
"idempotencyKey": {
|
||||
"type": "string",
|
||||
"minLength": 8,
|
||||
"description": "Deterministic key used to deduplicate events downstream."
|
||||
},
|
||||
"correlationId": {
|
||||
"type": "string",
|
||||
"description": "Correlation identifier that ties this event to a request or workflow."
|
||||
},
|
||||
"traceId": {
|
||||
"type": "string",
|
||||
"description": "W3C trace ID (32 hex chars) for distributed tracing."
|
||||
},
|
||||
"spanId": {
|
||||
"type": "string",
|
||||
"description": "Optional span identifier associated with traceId."
|
||||
},
|
||||
"scope": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["repo", "digest"],
|
||||
"properties": {
|
||||
"namespace": {"type": "string"},
|
||||
"repo": {"type": "string"},
|
||||
"digest": {"type": "string"},
|
||||
"component": {"type": "string"},
|
||||
"image": {"type": "string"}
|
||||
}
|
||||
},
|
||||
"attributes": {
|
||||
"type": "object",
|
||||
"description": "String attributes for downstream correlation (policy revision, scan id, etc.).",
|
||||
"additionalProperties": {"type": "string"}
|
||||
},
|
||||
"payload": {
|
||||
"type": "object",
|
||||
"additionalProperties": true,
|
||||
"required": ["reportId", "verdict", "summary", "links", "report"],
|
||||
"properties": {
|
||||
"reportId": {"type": "string"},
|
||||
"scanId": {"type": "string"},
|
||||
"imageDigest": {"type": "string"},
|
||||
"generatedAt": {"type": "string", "format": "date-time"},
|
||||
"verdict": {"enum": ["pass", "warn", "fail"]},
|
||||
"summary": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["total", "blocked", "warned", "ignored", "quieted"],
|
||||
"properties": {
|
||||
"total": {"type": "integer", "minimum": 0},
|
||||
"blocked": {"type": "integer", "minimum": 0},
|
||||
"warned": {"type": "integer", "minimum": 0},
|
||||
"ignored": {"type": "integer", "minimum": 0},
|
||||
"quieted": {"type": "integer", "minimum": 0}
|
||||
}
|
||||
},
|
||||
"delta": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"properties": {
|
||||
"newCritical": {"type": "integer", "minimum": 0},
|
||||
"newHigh": {"type": "integer", "minimum": 0},
|
||||
"kev": {
|
||||
"type": "array",
|
||||
"items": {"type": "string"}
|
||||
}
|
||||
}
|
||||
},
|
||||
"quietedFindingCount": {
|
||||
"type": "integer",
|
||||
"minimum": 0
|
||||
},
|
||||
"policy": {
|
||||
"type": "object",
|
||||
"description": "Policy revision metadata surfaced alongside the report."
|
||||
},
|
||||
"links": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"properties": {
|
||||
"ui": {"type": "string", "format": "uri"},
|
||||
"report": {"type": "string", "format": "uri"},
|
||||
"policy": {"type": "string", "format": "uri"},
|
||||
"attestation": {"type": "string", "format": "uri"}
|
||||
}
|
||||
},
|
||||
"dsse": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["payloadType", "payload", "signatures"],
|
||||
"properties": {
|
||||
"payloadType": {"type": "string"},
|
||||
"payload": {"type": "string"},
|
||||
"signatures": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["keyId", "algorithm", "signature"],
|
||||
"properties": {
|
||||
"keyId": {"type": "string"},
|
||||
"algorithm": {"type": "string"},
|
||||
"signature": {"type": "string"}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"report": {
|
||||
"type": "object",
|
||||
"description": "Canonical scanner report document that aligns with the DSSE payload."
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
{
|
||||
"$id": "https://stella-ops.org/schemas/events/scanner.event.report.ready@1.json",
|
||||
"$schema": "http://json-schema.org/draft-07/schema#",
|
||||
"title": "Scanner orchestrator event – report ready (v1)",
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": [
|
||||
"eventId",
|
||||
"kind",
|
||||
"version",
|
||||
"tenant",
|
||||
"occurredAt",
|
||||
"source",
|
||||
"idempotencyKey",
|
||||
"payload"
|
||||
],
|
||||
"properties": {
|
||||
"eventId": {
|
||||
"type": "string",
|
||||
"format": "uuid",
|
||||
"description": "Globally unique identifier for this occurrence."
|
||||
},
|
||||
"kind": {
|
||||
"const": "scanner.event.report.ready",
|
||||
"description": "Event kind identifier consumed by orchestrator subscribers."
|
||||
},
|
||||
"version": {
|
||||
"const": 1,
|
||||
"description": "Schema version for orchestrator envelopes."
|
||||
},
|
||||
"tenant": {
|
||||
"type": "string",
|
||||
"description": "Tenant that owns the scan/report."
|
||||
},
|
||||
"occurredAt": {
|
||||
"type": "string",
|
||||
"format": "date-time",
|
||||
"description": "Timestamp (UTC) when the report transitioned to ready."
|
||||
},
|
||||
"recordedAt": {
|
||||
"type": "string",
|
||||
"format": "date-time",
|
||||
"description": "Timestamp (UTC) when the event was persisted. Optional."
|
||||
},
|
||||
"source": {
|
||||
"type": "string",
|
||||
"description": "Producer identifier, e.g. `scanner.webservice`."
|
||||
},
|
||||
"idempotencyKey": {
|
||||
"type": "string",
|
||||
"minLength": 8,
|
||||
"description": "Deterministic key used to deduplicate events downstream."
|
||||
},
|
||||
"correlationId": {
|
||||
"type": "string",
|
||||
"description": "Correlation identifier that ties this event to a request or workflow."
|
||||
},
|
||||
"traceId": {
|
||||
"type": "string",
|
||||
"description": "W3C trace ID (32 hex chars) for distributed tracing."
|
||||
},
|
||||
"spanId": {
|
||||
"type": "string",
|
||||
"description": "Optional span identifier associated with traceId."
|
||||
},
|
||||
"scope": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["repo", "digest"],
|
||||
"properties": {
|
||||
"namespace": {"type": "string"},
|
||||
"repo": {"type": "string"},
|
||||
"digest": {"type": "string"},
|
||||
"component": {"type": "string"},
|
||||
"image": {"type": "string"}
|
||||
}
|
||||
},
|
||||
"attributes": {
|
||||
"type": "object",
|
||||
"description": "String attributes for downstream correlation (policy revision, scan id, etc.).",
|
||||
"additionalProperties": {"type": "string"}
|
||||
},
|
||||
"payload": {
|
||||
"type": "object",
|
||||
"additionalProperties": true,
|
||||
"required": ["reportId", "verdict", "summary", "links", "report"],
|
||||
"properties": {
|
||||
"reportId": {"type": "string"},
|
||||
"scanId": {"type": "string"},
|
||||
"imageDigest": {"type": "string"},
|
||||
"generatedAt": {"type": "string", "format": "date-time"},
|
||||
"verdict": {"enum": ["pass", "warn", "fail"]},
|
||||
"summary": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["total", "blocked", "warned", "ignored", "quieted"],
|
||||
"properties": {
|
||||
"total": {"type": "integer", "minimum": 0},
|
||||
"blocked": {"type": "integer", "minimum": 0},
|
||||
"warned": {"type": "integer", "minimum": 0},
|
||||
"ignored": {"type": "integer", "minimum": 0},
|
||||
"quieted": {"type": "integer", "minimum": 0}
|
||||
}
|
||||
},
|
||||
"delta": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"properties": {
|
||||
"newCritical": {"type": "integer", "minimum": 0},
|
||||
"newHigh": {"type": "integer", "minimum": 0},
|
||||
"kev": {
|
||||
"type": "array",
|
||||
"items": {"type": "string"}
|
||||
}
|
||||
}
|
||||
},
|
||||
"quietedFindingCount": {
|
||||
"type": "integer",
|
||||
"minimum": 0
|
||||
},
|
||||
"policy": {
|
||||
"type": "object",
|
||||
"description": "Policy revision metadata surfaced alongside the report."
|
||||
},
|
||||
"links": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"properties": {
|
||||
"ui": {"type": "string", "format": "uri"},
|
||||
"report": {"type": "string", "format": "uri"},
|
||||
"policy": {"type": "string", "format": "uri"},
|
||||
"attestation": {"type": "string", "format": "uri"}
|
||||
}
|
||||
},
|
||||
"dsse": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["payloadType", "payload", "signatures"],
|
||||
"properties": {
|
||||
"payloadType": {"type": "string"},
|
||||
"payload": {"type": "string"},
|
||||
"signatures": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["keyId", "algorithm", "signature"],
|
||||
"properties": {
|
||||
"keyId": {"type": "string"},
|
||||
"algorithm": {"type": "string"},
|
||||
"signature": {"type": "string"}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"report": {
|
||||
"type": "object",
|
||||
"description": "Canonical scanner report document that aligns with the DSSE payload."
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1,174 +1,174 @@
|
||||
{
|
||||
"$id": "https://stella-ops.org/schemas/events/scanner.event.scan.completed@1.json",
|
||||
"$schema": "http://json-schema.org/draft-07/schema#",
|
||||
"title": "Scanner orchestrator event – scan completed (v1)",
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": [
|
||||
"eventId",
|
||||
"kind",
|
||||
"version",
|
||||
"tenant",
|
||||
"occurredAt",
|
||||
"source",
|
||||
"idempotencyKey",
|
||||
"payload"
|
||||
],
|
||||
"properties": {
|
||||
"eventId": {
|
||||
"type": "string",
|
||||
"format": "uuid",
|
||||
"description": "Globally unique identifier for this occurrence."
|
||||
},
|
||||
"kind": {
|
||||
"const": "scanner.event.scan.completed",
|
||||
"description": "Event kind identifier consumed by orchestrator subscribers."
|
||||
},
|
||||
"version": {
|
||||
"const": 1,
|
||||
"description": "Schema version for orchestrator envelopes."
|
||||
},
|
||||
"tenant": {
|
||||
"type": "string",
|
||||
"description": "Tenant that owns the scan."
|
||||
},
|
||||
"occurredAt": {
|
||||
"type": "string",
|
||||
"format": "date-time",
|
||||
"description": "Timestamp (UTC) when the scan completed."
|
||||
},
|
||||
"recordedAt": {
|
||||
"type": "string",
|
||||
"format": "date-time",
|
||||
"description": "Timestamp (UTC) when the event was persisted. Optional."
|
||||
},
|
||||
"source": {
|
||||
"type": "string",
|
||||
"description": "Producer identifier, e.g. `scanner.webservice`."
|
||||
},
|
||||
"idempotencyKey": {
|
||||
"type": "string",
|
||||
"minLength": 8,
|
||||
"description": "Deterministic key used to deduplicate events downstream."
|
||||
},
|
||||
"correlationId": {
|
||||
"type": "string",
|
||||
"description": "Correlation identifier tying this event to a request or workflow."
|
||||
},
|
||||
"traceId": {
|
||||
"type": "string",
|
||||
"description": "W3C trace ID (32 hex chars) for distributed tracing."
|
||||
},
|
||||
"spanId": {
|
||||
"type": "string",
|
||||
"description": "Optional span identifier associated with traceId."
|
||||
},
|
||||
"scope": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["repo", "digest"],
|
||||
"properties": {
|
||||
"namespace": {"type": "string"},
|
||||
"repo": {"type": "string"},
|
||||
"digest": {"type": "string"},
|
||||
"component": {"type": "string"},
|
||||
"image": {"type": "string"}
|
||||
}
|
||||
},
|
||||
"attributes": {
|
||||
"type": "object",
|
||||
"description": "String attributes for downstream correlation (policy revision, scan id, etc.).",
|
||||
"additionalProperties": {"type": "string"}
|
||||
},
|
||||
"payload": {
|
||||
"type": "object",
|
||||
"additionalProperties": true,
|
||||
"required": ["reportId", "scanId", "imageDigest", "verdict", "summary", "report"],
|
||||
"properties": {
|
||||
"reportId": {"type": "string"},
|
||||
"scanId": {"type": "string"},
|
||||
"imageDigest": {"type": "string"},
|
||||
"verdict": {"enum": ["pass", "warn", "fail"]},
|
||||
"summary": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["total", "blocked", "warned", "ignored", "quieted"],
|
||||
"properties": {
|
||||
"total": {"type": "integer", "minimum": 0},
|
||||
"blocked": {"type": "integer", "minimum": 0},
|
||||
"warned": {"type": "integer", "minimum": 0},
|
||||
"ignored": {"type": "integer", "minimum": 0},
|
||||
"quieted": {"type": "integer", "minimum": 0}
|
||||
}
|
||||
},
|
||||
"delta": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"properties": {
|
||||
"newCritical": {"type": "integer", "minimum": 0},
|
||||
"newHigh": {"type": "integer", "minimum": 0},
|
||||
"kev": {
|
||||
"type": "array",
|
||||
"items": {"type": "string"}
|
||||
}
|
||||
}
|
||||
},
|
||||
"policy": {
|
||||
"type": "object",
|
||||
"description": "Policy revision metadata surfaced alongside the report."
|
||||
},
|
||||
"findings": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["id"],
|
||||
"properties": {
|
||||
"id": {"type": "string"},
|
||||
"severity": {"type": "string"},
|
||||
"cve": {"type": "string"},
|
||||
"purl": {"type": "string"},
|
||||
"reachability": {"type": "string"}
|
||||
}
|
||||
}
|
||||
},
|
||||
"links": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"properties": {
|
||||
"ui": {"type": "string", "format": "uri"},
|
||||
"report": {"type": "string", "format": "uri"},
|
||||
"policy": {"type": "string", "format": "uri"},
|
||||
"attestation": {"type": "string", "format": "uri"}
|
||||
}
|
||||
},
|
||||
"dsse": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["payloadType", "payload", "signatures"],
|
||||
"properties": {
|
||||
"payloadType": {"type": "string"},
|
||||
"payload": {"type": "string"},
|
||||
"signatures": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["keyId", "algorithm", "signature"],
|
||||
"properties": {
|
||||
"keyId": {"type": "string"},
|
||||
"algorithm": {"type": "string"},
|
||||
"signature": {"type": "string"}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"report": {
|
||||
"type": "object",
|
||||
"description": "Canonical scanner report document that aligns with the DSSE payload."
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
{
|
||||
"$id": "https://stella-ops.org/schemas/events/scanner.event.scan.completed@1.json",
|
||||
"$schema": "http://json-schema.org/draft-07/schema#",
|
||||
"title": "Scanner orchestrator event – scan completed (v1)",
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": [
|
||||
"eventId",
|
||||
"kind",
|
||||
"version",
|
||||
"tenant",
|
||||
"occurredAt",
|
||||
"source",
|
||||
"idempotencyKey",
|
||||
"payload"
|
||||
],
|
||||
"properties": {
|
||||
"eventId": {
|
||||
"type": "string",
|
||||
"format": "uuid",
|
||||
"description": "Globally unique identifier for this occurrence."
|
||||
},
|
||||
"kind": {
|
||||
"const": "scanner.event.scan.completed",
|
||||
"description": "Event kind identifier consumed by orchestrator subscribers."
|
||||
},
|
||||
"version": {
|
||||
"const": 1,
|
||||
"description": "Schema version for orchestrator envelopes."
|
||||
},
|
||||
"tenant": {
|
||||
"type": "string",
|
||||
"description": "Tenant that owns the scan."
|
||||
},
|
||||
"occurredAt": {
|
||||
"type": "string",
|
||||
"format": "date-time",
|
||||
"description": "Timestamp (UTC) when the scan completed."
|
||||
},
|
||||
"recordedAt": {
|
||||
"type": "string",
|
||||
"format": "date-time",
|
||||
"description": "Timestamp (UTC) when the event was persisted. Optional."
|
||||
},
|
||||
"source": {
|
||||
"type": "string",
|
||||
"description": "Producer identifier, e.g. `scanner.webservice`."
|
||||
},
|
||||
"idempotencyKey": {
|
||||
"type": "string",
|
||||
"minLength": 8,
|
||||
"description": "Deterministic key used to deduplicate events downstream."
|
||||
},
|
||||
"correlationId": {
|
||||
"type": "string",
|
||||
"description": "Correlation identifier tying this event to a request or workflow."
|
||||
},
|
||||
"traceId": {
|
||||
"type": "string",
|
||||
"description": "W3C trace ID (32 hex chars) for distributed tracing."
|
||||
},
|
||||
"spanId": {
|
||||
"type": "string",
|
||||
"description": "Optional span identifier associated with traceId."
|
||||
},
|
||||
"scope": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["repo", "digest"],
|
||||
"properties": {
|
||||
"namespace": {"type": "string"},
|
||||
"repo": {"type": "string"},
|
||||
"digest": {"type": "string"},
|
||||
"component": {"type": "string"},
|
||||
"image": {"type": "string"}
|
||||
}
|
||||
},
|
||||
"attributes": {
|
||||
"type": "object",
|
||||
"description": "String attributes for downstream correlation (policy revision, scan id, etc.).",
|
||||
"additionalProperties": {"type": "string"}
|
||||
},
|
||||
"payload": {
|
||||
"type": "object",
|
||||
"additionalProperties": true,
|
||||
"required": ["reportId", "scanId", "imageDigest", "verdict", "summary", "report"],
|
||||
"properties": {
|
||||
"reportId": {"type": "string"},
|
||||
"scanId": {"type": "string"},
|
||||
"imageDigest": {"type": "string"},
|
||||
"verdict": {"enum": ["pass", "warn", "fail"]},
|
||||
"summary": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["total", "blocked", "warned", "ignored", "quieted"],
|
||||
"properties": {
|
||||
"total": {"type": "integer", "minimum": 0},
|
||||
"blocked": {"type": "integer", "minimum": 0},
|
||||
"warned": {"type": "integer", "minimum": 0},
|
||||
"ignored": {"type": "integer", "minimum": 0},
|
||||
"quieted": {"type": "integer", "minimum": 0}
|
||||
}
|
||||
},
|
||||
"delta": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"properties": {
|
||||
"newCritical": {"type": "integer", "minimum": 0},
|
||||
"newHigh": {"type": "integer", "minimum": 0},
|
||||
"kev": {
|
||||
"type": "array",
|
||||
"items": {"type": "string"}
|
||||
}
|
||||
}
|
||||
},
|
||||
"policy": {
|
||||
"type": "object",
|
||||
"description": "Policy revision metadata surfaced alongside the report."
|
||||
},
|
||||
"findings": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["id"],
|
||||
"properties": {
|
||||
"id": {"type": "string"},
|
||||
"severity": {"type": "string"},
|
||||
"cve": {"type": "string"},
|
||||
"purl": {"type": "string"},
|
||||
"reachability": {"type": "string"}
|
||||
}
|
||||
}
|
||||
},
|
||||
"links": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"properties": {
|
||||
"ui": {"type": "string", "format": "uri"},
|
||||
"report": {"type": "string", "format": "uri"},
|
||||
"policy": {"type": "string", "format": "uri"},
|
||||
"attestation": {"type": "string", "format": "uri"}
|
||||
}
|
||||
},
|
||||
"dsse": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["payloadType", "payload", "signatures"],
|
||||
"properties": {
|
||||
"payloadType": {"type": "string"},
|
||||
"payload": {"type": "string"},
|
||||
"signatures": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["keyId", "algorithm", "signature"],
|
||||
"properties": {
|
||||
"keyId": {"type": "string"},
|
||||
"algorithm": {"type": "string"},
|
||||
"signature": {"type": "string"}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"report": {
|
||||
"type": "object",
|
||||
"description": "Canonical scanner report document that aligns with the DSSE payload."
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1,196 +1,196 @@
|
||||
{
|
||||
"$id": "https://stella-ops.org/schemas/events/scheduler.graph.job.completed@1.json",
|
||||
"$schema": "http://json-schema.org/draft-07/schema#",
|
||||
"title": "Scheduler Graph Job Completed Event",
|
||||
"description": "Legacy scheduler event emitted when a graph build or overlay job reaches a terminal state. Consumers validate downstream caches and surface overlay freshness.",
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["eventId", "kind", "tenant", "ts", "payload"],
|
||||
"properties": {
|
||||
"eventId": {
|
||||
"type": "string",
|
||||
"format": "uuid",
|
||||
"description": "Globally unique identifier per event."
|
||||
},
|
||||
"kind": {
|
||||
"const": "scheduler.graph.job.completed"
|
||||
},
|
||||
"tenant": {
|
||||
"type": "string",
|
||||
"description": "Tenant identifier scoped to the originating job."
|
||||
},
|
||||
"ts": {
|
||||
"type": "string",
|
||||
"format": "date-time",
|
||||
"description": "UTC timestamp when the job reached a terminal state."
|
||||
},
|
||||
"payload": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["jobType", "job", "status", "occurredAt"],
|
||||
"properties": {
|
||||
"jobType": {
|
||||
"type": "string",
|
||||
"enum": ["build", "overlay"],
|
||||
"description": "Job flavour, matches the CLR type of the serialized job payload."
|
||||
},
|
||||
"status": {
|
||||
"type": "string",
|
||||
"enum": ["completed", "failed", "cancelled"],
|
||||
"description": "Terminal status recorded for the job."
|
||||
},
|
||||
"occurredAt": {
|
||||
"type": "string",
|
||||
"format": "date-time",
|
||||
"description": "UTC timestamp of the terminal transition, mirrors job.CompletedAt."
|
||||
},
|
||||
"job": {
|
||||
"oneOf": [
|
||||
{"$ref": "#/definitions/graphBuildJob"},
|
||||
{"$ref": "#/definitions/graphOverlayJob"}
|
||||
],
|
||||
"description": "Canonical serialized representation of the finished job."
|
||||
},
|
||||
"resultUri": {
|
||||
"type": "string",
|
||||
"description": "Optional URI pointing to Cartographer snapshot or overlay bundle (if available)."
|
||||
}
|
||||
}
|
||||
},
|
||||
"attributes": {
|
||||
"type": "object",
|
||||
"description": "Optional correlation bag for downstream consumers.",
|
||||
"additionalProperties": {
|
||||
"type": "string"
|
||||
}
|
||||
}
|
||||
},
|
||||
"definitions": {
|
||||
"graphBuildJob": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": [
|
||||
"schemaVersion",
|
||||
"id",
|
||||
"tenantId",
|
||||
"sbomId",
|
||||
"sbomVersionId",
|
||||
"sbomDigest",
|
||||
"status",
|
||||
"trigger",
|
||||
"attempts",
|
||||
"createdAt"
|
||||
],
|
||||
"properties": {
|
||||
"schemaVersion": {
|
||||
"const": "scheduler.graph-build-job@1"
|
||||
},
|
||||
"id": {"type": "string"},
|
||||
"tenantId": {"type": "string"},
|
||||
"sbomId": {"type": "string"},
|
||||
"sbomVersionId": {"type": "string"},
|
||||
"sbomDigest": {
|
||||
"type": "string",
|
||||
"pattern": "^sha256:[a-f0-9]{64}$"
|
||||
},
|
||||
"graphSnapshotId": {"type": "string"},
|
||||
"status": {
|
||||
"type": "string",
|
||||
"enum": ["pending", "queued", "running", "completed", "failed", "cancelled"]
|
||||
},
|
||||
"trigger": {
|
||||
"type": "string",
|
||||
"enum": ["sbom-version", "backfill", "manual"]
|
||||
},
|
||||
"attempts": {
|
||||
"type": "integer",
|
||||
"minimum": 0
|
||||
},
|
||||
"cartographerJobId": {"type": "string"},
|
||||
"correlationId": {"type": "string"},
|
||||
"createdAt": {
|
||||
"type": "string",
|
||||
"format": "date-time"
|
||||
},
|
||||
"startedAt": {
|
||||
"type": "string",
|
||||
"format": "date-time"
|
||||
},
|
||||
"completedAt": {
|
||||
"type": "string",
|
||||
"format": "date-time"
|
||||
},
|
||||
"error": {"type": "string"},
|
||||
"metadata": {
|
||||
"type": "object",
|
||||
"additionalProperties": {"type": "string"}
|
||||
}
|
||||
}
|
||||
},
|
||||
"graphOverlayJob": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": [
|
||||
"schemaVersion",
|
||||
"id",
|
||||
"tenantId",
|
||||
"graphSnapshotId",
|
||||
"overlayKind",
|
||||
"overlayKey",
|
||||
"status",
|
||||
"trigger",
|
||||
"attempts",
|
||||
"createdAt"
|
||||
],
|
||||
"properties": {
|
||||
"schemaVersion": {
|
||||
"const": "scheduler.graph-overlay-job@1"
|
||||
},
|
||||
"id": {"type": "string"},
|
||||
"tenantId": {"type": "string"},
|
||||
"graphSnapshotId": {"type": "string"},
|
||||
"buildJobId": {"type": "string"},
|
||||
"overlayKind": {
|
||||
"type": "string",
|
||||
"enum": ["policy", "advisory", "vex"]
|
||||
},
|
||||
"overlayKey": {"type": "string"},
|
||||
"subjects": {
|
||||
"type": "array",
|
||||
"items": {"type": "string"},
|
||||
"uniqueItems": true
|
||||
},
|
||||
"status": {
|
||||
"type": "string",
|
||||
"enum": ["pending", "queued", "running", "completed", "failed", "cancelled"]
|
||||
},
|
||||
"trigger": {
|
||||
"type": "string",
|
||||
"enum": ["policy", "advisory", "vex", "sbom-version", "manual"]
|
||||
},
|
||||
"attempts": {
|
||||
"type": "integer",
|
||||
"minimum": 0
|
||||
},
|
||||
"correlationId": {"type": "string"},
|
||||
"createdAt": {
|
||||
"type": "string",
|
||||
"format": "date-time"
|
||||
},
|
||||
"startedAt": {
|
||||
"type": "string",
|
||||
"format": "date-time"
|
||||
},
|
||||
"completedAt": {
|
||||
"type": "string",
|
||||
"format": "date-time"
|
||||
},
|
||||
"error": {"type": "string"},
|
||||
"metadata": {
|
||||
"type": "object",
|
||||
"additionalProperties": {"type": "string"}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
{
|
||||
"$id": "https://stella-ops.org/schemas/events/scheduler.graph.job.completed@1.json",
|
||||
"$schema": "http://json-schema.org/draft-07/schema#",
|
||||
"title": "Scheduler Graph Job Completed Event",
|
||||
"description": "Legacy scheduler event emitted when a graph build or overlay job reaches a terminal state. Consumers validate downstream caches and surface overlay freshness.",
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["eventId", "kind", "tenant", "ts", "payload"],
|
||||
"properties": {
|
||||
"eventId": {
|
||||
"type": "string",
|
||||
"format": "uuid",
|
||||
"description": "Globally unique identifier per event."
|
||||
},
|
||||
"kind": {
|
||||
"const": "scheduler.graph.job.completed"
|
||||
},
|
||||
"tenant": {
|
||||
"type": "string",
|
||||
"description": "Tenant identifier scoped to the originating job."
|
||||
},
|
||||
"ts": {
|
||||
"type": "string",
|
||||
"format": "date-time",
|
||||
"description": "UTC timestamp when the job reached a terminal state."
|
||||
},
|
||||
"payload": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["jobType", "job", "status", "occurredAt"],
|
||||
"properties": {
|
||||
"jobType": {
|
||||
"type": "string",
|
||||
"enum": ["build", "overlay"],
|
||||
"description": "Job flavour, matches the CLR type of the serialized job payload."
|
||||
},
|
||||
"status": {
|
||||
"type": "string",
|
||||
"enum": ["completed", "failed", "cancelled"],
|
||||
"description": "Terminal status recorded for the job."
|
||||
},
|
||||
"occurredAt": {
|
||||
"type": "string",
|
||||
"format": "date-time",
|
||||
"description": "UTC timestamp of the terminal transition, mirrors job.CompletedAt."
|
||||
},
|
||||
"job": {
|
||||
"oneOf": [
|
||||
{"$ref": "#/definitions/graphBuildJob"},
|
||||
{"$ref": "#/definitions/graphOverlayJob"}
|
||||
],
|
||||
"description": "Canonical serialized representation of the finished job."
|
||||
},
|
||||
"resultUri": {
|
||||
"type": "string",
|
||||
"description": "Optional URI pointing to Cartographer snapshot or overlay bundle (if available)."
|
||||
}
|
||||
}
|
||||
},
|
||||
"attributes": {
|
||||
"type": "object",
|
||||
"description": "Optional correlation bag for downstream consumers.",
|
||||
"additionalProperties": {
|
||||
"type": "string"
|
||||
}
|
||||
}
|
||||
},
|
||||
"definitions": {
|
||||
"graphBuildJob": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": [
|
||||
"schemaVersion",
|
||||
"id",
|
||||
"tenantId",
|
||||
"sbomId",
|
||||
"sbomVersionId",
|
||||
"sbomDigest",
|
||||
"status",
|
||||
"trigger",
|
||||
"attempts",
|
||||
"createdAt"
|
||||
],
|
||||
"properties": {
|
||||
"schemaVersion": {
|
||||
"const": "scheduler.graph-build-job@1"
|
||||
},
|
||||
"id": {"type": "string"},
|
||||
"tenantId": {"type": "string"},
|
||||
"sbomId": {"type": "string"},
|
||||
"sbomVersionId": {"type": "string"},
|
||||
"sbomDigest": {
|
||||
"type": "string",
|
||||
"pattern": "^sha256:[a-f0-9]{64}$"
|
||||
},
|
||||
"graphSnapshotId": {"type": "string"},
|
||||
"status": {
|
||||
"type": "string",
|
||||
"enum": ["pending", "queued", "running", "completed", "failed", "cancelled"]
|
||||
},
|
||||
"trigger": {
|
||||
"type": "string",
|
||||
"enum": ["sbom-version", "backfill", "manual"]
|
||||
},
|
||||
"attempts": {
|
||||
"type": "integer",
|
||||
"minimum": 0
|
||||
},
|
||||
"cartographerJobId": {"type": "string"},
|
||||
"correlationId": {"type": "string"},
|
||||
"createdAt": {
|
||||
"type": "string",
|
||||
"format": "date-time"
|
||||
},
|
||||
"startedAt": {
|
||||
"type": "string",
|
||||
"format": "date-time"
|
||||
},
|
||||
"completedAt": {
|
||||
"type": "string",
|
||||
"format": "date-time"
|
||||
},
|
||||
"error": {"type": "string"},
|
||||
"metadata": {
|
||||
"type": "object",
|
||||
"additionalProperties": {"type": "string"}
|
||||
}
|
||||
}
|
||||
},
|
||||
"graphOverlayJob": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": [
|
||||
"schemaVersion",
|
||||
"id",
|
||||
"tenantId",
|
||||
"graphSnapshotId",
|
||||
"overlayKind",
|
||||
"overlayKey",
|
||||
"status",
|
||||
"trigger",
|
||||
"attempts",
|
||||
"createdAt"
|
||||
],
|
||||
"properties": {
|
||||
"schemaVersion": {
|
||||
"const": "scheduler.graph-overlay-job@1"
|
||||
},
|
||||
"id": {"type": "string"},
|
||||
"tenantId": {"type": "string"},
|
||||
"graphSnapshotId": {"type": "string"},
|
||||
"buildJobId": {"type": "string"},
|
||||
"overlayKind": {
|
||||
"type": "string",
|
||||
"enum": ["policy", "advisory", "vex"]
|
||||
},
|
||||
"overlayKey": {"type": "string"},
|
||||
"subjects": {
|
||||
"type": "array",
|
||||
"items": {"type": "string"},
|
||||
"uniqueItems": true
|
||||
},
|
||||
"status": {
|
||||
"type": "string",
|
||||
"enum": ["pending", "queued", "running", "completed", "failed", "cancelled"]
|
||||
},
|
||||
"trigger": {
|
||||
"type": "string",
|
||||
"enum": ["policy", "advisory", "vex", "sbom-version", "manual"]
|
||||
},
|
||||
"attempts": {
|
||||
"type": "integer",
|
||||
"minimum": 0
|
||||
},
|
||||
"correlationId": {"type": "string"},
|
||||
"createdAt": {
|
||||
"type": "string",
|
||||
"format": "date-time"
|
||||
},
|
||||
"startedAt": {
|
||||
"type": "string",
|
||||
"format": "date-time"
|
||||
},
|
||||
"completedAt": {
|
||||
"type": "string",
|
||||
"format": "date-time"
|
||||
},
|
||||
"error": {"type": "string"},
|
||||
"metadata": {
|
||||
"type": "object",
|
||||
"additionalProperties": {"type": "string"}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1,16 +1,16 @@
|
||||
# Policy Examples
|
||||
|
||||
Sample `stella-dsl@1` policies illustrating common deployment personas. Each example includes commentary, CLI usage hints, and a compliance checklist.
|
||||
|
||||
| Example | Description |
|
||||
|---------|-------------|
|
||||
| [Baseline](baseline.md) | Balanced production defaults (block critical, respect strong VEX). |
|
||||
| [Serverless](serverless.md) | Aggressive blocking for serverless workloads (no High+, pinned base images). |
|
||||
| [Internal Only](internal-only.md) | Lenient policy for internal/dev environments with KEV safeguards. |
|
||||
|
||||
Policy source files (`*.stella`) live alongside the documentation so you can copy/paste or use `stella policy new --from file://...`.
|
||||
|
||||
---
|
||||
|
||||
*Last updated: 2025-10-26.*
|
||||
|
||||
# Policy Examples
|
||||
|
||||
Sample `stella-dsl@1` policies illustrating common deployment personas. Each example includes commentary, CLI usage hints, and a compliance checklist.
|
||||
|
||||
| Example | Description |
|
||||
|---------|-------------|
|
||||
| [Baseline](baseline.md) | Balanced production defaults (block critical, respect strong VEX). |
|
||||
| [Serverless](serverless.md) | Aggressive blocking for serverless workloads (no High+, pinned base images). |
|
||||
| [Internal Only](internal-only.md) | Lenient policy for internal/dev environments with KEV safeguards. |
|
||||
|
||||
Policy source files (`*.stella`) live alongside the documentation so you can copy/paste or use `stella policy new --from file://...`.
|
||||
|
||||
---
|
||||
|
||||
*Last updated: 2025-10-26.*
|
||||
|
||||
|
||||
@@ -1,79 +1,79 @@
|
||||
# Baseline Policy Example (`baseline.stella`)
|
||||
|
||||
This sample policy provides a balanced default for production workloads: block critical findings, require strong VEX justifications to suppress advisories, and warn on deprecated runtimes. Use it as a starting point for tenants that want guardrails without excessive noise.
|
||||
|
||||
```dsl
|
||||
policy "Baseline Production Policy" syntax "stella-dsl@1" {
|
||||
metadata {
|
||||
description = "Block critical, escalate high, enforce VEX justifications."
|
||||
tags = ["baseline","production"]
|
||||
}
|
||||
|
||||
profile severity {
|
||||
map vendor_weight {
|
||||
source "GHSA" => +0.5
|
||||
source "OSV" => +0.0
|
||||
source "VendorX" => -0.2
|
||||
}
|
||||
env exposure_adjustments {
|
||||
if env.exposure == "internet" then +0.5
|
||||
if env.runtime == "legacy" then +0.3
|
||||
}
|
||||
}
|
||||
|
||||
rule block_critical priority 5 {
|
||||
when severity.normalized >= "Critical"
|
||||
then status := "blocked"
|
||||
because "Critical severity must be remediated before deploy."
|
||||
}
|
||||
|
||||
rule escalate_high_internet {
|
||||
when severity.normalized == "High"
|
||||
and env.exposure == "internet"
|
||||
then escalate to severity_band("Critical")
|
||||
because "High severity on internet-exposed asset escalates to critical."
|
||||
}
|
||||
|
||||
rule require_vex_justification {
|
||||
when vex.any(status in ["not_affected","fixed"])
|
||||
and vex.justification in ["component_not_present","vulnerable_code_not_present"]
|
||||
then status := vex.status
|
||||
annotate winning_statement := vex.latest().statementId
|
||||
because "Respect strong vendor VEX claims."
|
||||
}
|
||||
|
||||
rule alert_warn_eol_runtime priority 1 {
|
||||
when severity.normalized <= "Medium"
|
||||
and sbom.has_tag("runtime:eol")
|
||||
then warn message "Runtime marked as EOL; upgrade recommended."
|
||||
because "Deprecated runtime should be upgraded."
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Commentary
|
||||
|
||||
- **Severity profile** tightens vendor weights and applies exposure modifiers so internet-facing/high severity pairs escalate automatically.
|
||||
- **VEX rule** only honours strong justifications, preventing weaker claims from hiding issues.
|
||||
- **Warnings first** – The `alert_warn_eol_runtime` rule name ensures it sorts before the require-VEX rule, keeping alerts visible without flipping to `RequiresVex`.
|
||||
- Works well as shared `tenant-global` baseline; use tenant overrides for stricter tolerant environments.
|
||||
|
||||
## Try it out
|
||||
|
||||
```bash
|
||||
stella policy new --policy-id P-baseline --template blank --open
|
||||
stella policy lint examples/policies/baseline.stella
|
||||
stella policy simulate P-baseline --candidate 1 --sbom sbom:sample-prod
|
||||
```
|
||||
|
||||
## Compliance checklist
|
||||
|
||||
- [ ] Policy compiled via `stella policy lint` without diagnostics.
|
||||
- [ ] Simulation diff reviewed against golden SBOM set.
|
||||
- [ ] Approval note documents rationale before promoting to production.
|
||||
- [ ] EOL runtime tags kept up to date in SBOM metadata.
|
||||
- [ ] VEX vendor allow-list reviewed quarterly.
|
||||
|
||||
---
|
||||
|
||||
*Last updated: 2025-10-26.*
|
||||
# Baseline Policy Example (`baseline.stella`)
|
||||
|
||||
This sample policy provides a balanced default for production workloads: block critical findings, require strong VEX justifications to suppress advisories, and warn on deprecated runtimes. Use it as a starting point for tenants that want guardrails without excessive noise.
|
||||
|
||||
```dsl
|
||||
policy "Baseline Production Policy" syntax "stella-dsl@1" {
|
||||
metadata {
|
||||
description = "Block critical, escalate high, enforce VEX justifications."
|
||||
tags = ["baseline","production"]
|
||||
}
|
||||
|
||||
profile severity {
|
||||
map vendor_weight {
|
||||
source "GHSA" => +0.5
|
||||
source "OSV" => +0.0
|
||||
source "VendorX" => -0.2
|
||||
}
|
||||
env exposure_adjustments {
|
||||
if env.exposure == "internet" then +0.5
|
||||
if env.runtime == "legacy" then +0.3
|
||||
}
|
||||
}
|
||||
|
||||
rule block_critical priority 5 {
|
||||
when severity.normalized >= "Critical"
|
||||
then status := "blocked"
|
||||
because "Critical severity must be remediated before deploy."
|
||||
}
|
||||
|
||||
rule escalate_high_internet {
|
||||
when severity.normalized == "High"
|
||||
and env.exposure == "internet"
|
||||
then escalate to severity_band("Critical")
|
||||
because "High severity on internet-exposed asset escalates to critical."
|
||||
}
|
||||
|
||||
rule require_vex_justification {
|
||||
when vex.any(status in ["not_affected","fixed"])
|
||||
and vex.justification in ["component_not_present","vulnerable_code_not_present"]
|
||||
then status := vex.status
|
||||
annotate winning_statement := vex.latest().statementId
|
||||
because "Respect strong vendor VEX claims."
|
||||
}
|
||||
|
||||
rule alert_warn_eol_runtime priority 1 {
|
||||
when severity.normalized <= "Medium"
|
||||
and sbom.has_tag("runtime:eol")
|
||||
then warn message "Runtime marked as EOL; upgrade recommended."
|
||||
because "Deprecated runtime should be upgraded."
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Commentary
|
||||
|
||||
- **Severity profile** tightens vendor weights and applies exposure modifiers so internet-facing/high severity pairs escalate automatically.
|
||||
- **VEX rule** only honours strong justifications, preventing weaker claims from hiding issues.
|
||||
- **Warnings first** – The `alert_warn_eol_runtime` rule name ensures it sorts before the require-VEX rule, keeping alerts visible without flipping to `RequiresVex`.
|
||||
- Works well as shared `tenant-global` baseline; use tenant overrides for stricter tolerant environments.
|
||||
|
||||
## Try it out
|
||||
|
||||
```bash
|
||||
stella policy new --policy-id P-baseline --template blank --open
|
||||
stella policy lint examples/policies/baseline.stella
|
||||
stella policy simulate P-baseline --candidate 1 --sbom sbom:sample-prod
|
||||
```
|
||||
|
||||
## Compliance checklist
|
||||
|
||||
- [ ] Policy compiled via `stella policy lint` without diagnostics.
|
||||
- [ ] Simulation diff reviewed against golden SBOM set.
|
||||
- [ ] Approval note documents rationale before promoting to production.
|
||||
- [ ] EOL runtime tags kept up to date in SBOM metadata.
|
||||
- [ ] VEX vendor allow-list reviewed quarterly.
|
||||
|
||||
---
|
||||
|
||||
*Last updated: 2025-10-26.*
|
||||
|
||||
@@ -1,46 +1,46 @@
|
||||
policy "Baseline Production Policy" syntax "stella-dsl@1" {
|
||||
metadata {
|
||||
description = "Block critical, escalate high, enforce VEX justifications."
|
||||
tags = ["baseline","production"]
|
||||
}
|
||||
|
||||
profile severity {
|
||||
map vendor_weight {
|
||||
source "GHSA" => +0.5
|
||||
source "OSV" => +0.0
|
||||
source "VendorX" => -0.2
|
||||
}
|
||||
env exposure_adjustments {
|
||||
if env.exposure == "internet" then +0.5
|
||||
if env.runtime == "legacy" then +0.3
|
||||
}
|
||||
}
|
||||
|
||||
rule block_critical priority 5 {
|
||||
when severity.normalized >= "Critical"
|
||||
then status := "blocked"
|
||||
because "Critical severity must be remediated before deploy."
|
||||
}
|
||||
|
||||
rule escalate_high_internet {
|
||||
when severity.normalized == "High"
|
||||
and env.exposure == "internet"
|
||||
then escalate to severity_band("Critical")
|
||||
because "High severity on internet-exposed asset escalates to critical."
|
||||
}
|
||||
|
||||
rule require_vex_justification {
|
||||
when vex.any(status in ["not_affected","fixed"])
|
||||
and vex.justification in ["component_not_present","vulnerable_code_not_present"]
|
||||
then status := vex.status
|
||||
annotate winning_statement := vex.latest().statementId
|
||||
because "Respect strong vendor VEX claims."
|
||||
}
|
||||
|
||||
rule alert_warn_eol_runtime priority 1 {
|
||||
when severity.normalized <= "Medium"
|
||||
and sbom.has_tag("runtime:eol")
|
||||
then warn message "Runtime marked as EOL; upgrade recommended."
|
||||
because "Deprecated runtime should be upgraded."
|
||||
}
|
||||
}
|
||||
policy "Baseline Production Policy" syntax "stella-dsl@1" {
|
||||
metadata {
|
||||
description = "Block critical, escalate high, enforce VEX justifications."
|
||||
tags = ["baseline","production"]
|
||||
}
|
||||
|
||||
profile severity {
|
||||
map vendor_weight {
|
||||
source "GHSA" => +0.5
|
||||
source "OSV" => +0.0
|
||||
source "VendorX" => -0.2
|
||||
}
|
||||
env exposure_adjustments {
|
||||
if env.exposure == "internet" then +0.5
|
||||
if env.runtime == "legacy" then +0.3
|
||||
}
|
||||
}
|
||||
|
||||
rule block_critical priority 5 {
|
||||
when severity.normalized >= "Critical"
|
||||
then status := "blocked"
|
||||
because "Critical severity must be remediated before deploy."
|
||||
}
|
||||
|
||||
rule escalate_high_internet {
|
||||
when severity.normalized == "High"
|
||||
and env.exposure == "internet"
|
||||
then escalate to severity_band("Critical")
|
||||
because "High severity on internet-exposed asset escalates to critical."
|
||||
}
|
||||
|
||||
rule require_vex_justification {
|
||||
when vex.any(status in ["not_affected","fixed"])
|
||||
and vex.justification in ["component_not_present","vulnerable_code_not_present"]
|
||||
then status := vex.status
|
||||
annotate winning_statement := vex.latest().statementId
|
||||
because "Respect strong vendor VEX claims."
|
||||
}
|
||||
|
||||
rule alert_warn_eol_runtime priority 1 {
|
||||
when severity.normalized <= "Medium"
|
||||
and sbom.has_tag("runtime:eol")
|
||||
then warn message "Runtime marked as EOL; upgrade recommended."
|
||||
because "Deprecated runtime should be upgraded."
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1,34 +1,34 @@
|
||||
version: "1.0"
|
||||
metadata:
|
||||
description: Baseline production policy
|
||||
tags:
|
||||
- baseline
|
||||
- production
|
||||
rules:
|
||||
- name: Block Critical
|
||||
severity: [Critical]
|
||||
action: block
|
||||
|
||||
- name: Escalate High Internet
|
||||
severity: [High]
|
||||
environments: [internet]
|
||||
action:
|
||||
type: escalate
|
||||
escalate:
|
||||
minimumSeverity: Critical
|
||||
|
||||
- name: Require VEX justification
|
||||
sources: [NVD, GHSA]
|
||||
action:
|
||||
type: requireVex
|
||||
requireVex:
|
||||
vendors: [VendorX, VendorY]
|
||||
justifications:
|
||||
- component_not_present
|
||||
- vulnerable_code_not_present
|
||||
|
||||
- name: Alert warn EOL runtime
|
||||
priority: 1
|
||||
severity: [Low, Medium]
|
||||
tags: [runtime:eol]
|
||||
action: warn
|
||||
version: "1.0"
|
||||
metadata:
|
||||
description: Baseline production policy
|
||||
tags:
|
||||
- baseline
|
||||
- production
|
||||
rules:
|
||||
- name: Block Critical
|
||||
severity: [Critical]
|
||||
action: block
|
||||
|
||||
- name: Escalate High Internet
|
||||
severity: [High]
|
||||
environments: [internet]
|
||||
action:
|
||||
type: escalate
|
||||
escalate:
|
||||
minimumSeverity: Critical
|
||||
|
||||
- name: Require VEX justification
|
||||
sources: [NVD, GHSA]
|
||||
action:
|
||||
type: requireVex
|
||||
requireVex:
|
||||
vendors: [VendorX, VendorY]
|
||||
justifications:
|
||||
- component_not_present
|
||||
- vulnerable_code_not_present
|
||||
|
||||
- name: Alert warn EOL runtime
|
||||
priority: 1
|
||||
severity: [Low, Medium]
|
||||
tags: [runtime:eol]
|
||||
action: warn
|
||||
|
||||
@@ -1,72 +1,72 @@
|
||||
# Internal-Only Policy Example (`internal-only.stella`)
|
||||
|
||||
A relaxed profile for internal services and development environments: allow Medium severities with warnings, rely on VEX more heavily, but still block KEV/actively exploited advisories.
|
||||
|
||||
```dsl
|
||||
policy "Internal Only Policy" syntax "stella-dsl@1" {
|
||||
metadata {
|
||||
description = "Lenient policy for internal / dev tenants."
|
||||
tags = ["internal","dev"]
|
||||
}
|
||||
|
||||
profile severity {
|
||||
env exposure_adjustments {
|
||||
if env.exposure == "internal" then -0.4
|
||||
if env.stage == "dev" then -0.6
|
||||
}
|
||||
}
|
||||
|
||||
rule block_kev priority 1 {
|
||||
when advisory.has_tag("kev")
|
||||
then status := "blocked"
|
||||
because "Known exploited vulnerabilities must be remediated."
|
||||
}
|
||||
|
||||
rule allow_medium_with_warning {
|
||||
when severity.normalized == "Medium"
|
||||
and env.exposure == "internal"
|
||||
then warn message "Medium severity permitted in internal environments."
|
||||
because "Allow Medium findings with warning for internal workloads."
|
||||
}
|
||||
|
||||
rule accept_vendor_vex {
|
||||
when vex.any(status in ["not_affected","fixed"])
|
||||
then status := vex.status
|
||||
annotate justification := vex.latest().justification
|
||||
because "Trust vendor VEX statements for internal scope."
|
||||
}
|
||||
|
||||
rule quiet_low_priority {
|
||||
when severity.normalized <= "Low"
|
||||
then ignore until "2026-01-01T00:00:00Z"
|
||||
because "Quiet low severity until next annual remediation sweep."
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Commentary
|
||||
|
||||
- Suitable for staging/dev tenants with lower blast radius.
|
||||
- KEV advisories override lenient behaviour to maintain minimum security bar.
|
||||
- Warnings ensure Medium findings stay visible in dashboards and CLI outputs.
|
||||
- Quiet rule enforces planned clean-up date; update before expiry.
|
||||
|
||||
## Try it out
|
||||
|
||||
```bash
|
||||
stella policy lint examples/policies/internal-only.stella
|
||||
stella policy simulate P-internal --candidate 1 \
|
||||
--sbom sbom:internal-service --env exposure=internal --env stage=dev
|
||||
```
|
||||
|
||||
## Compliance checklist
|
||||
|
||||
- [ ] Tenant classified as internal-only with documented risk acceptance.
|
||||
- [ ] KEV feed synced (Concelier) and tags confirmed before relying on rule.
|
||||
- [ ] Quiet expiry tracked; remediation backlog updated prior to deadline.
|
||||
- [ ] Developers informed that warnings still affect quality score.
|
||||
- [ ] Policy not used for production or internet-exposed services.
|
||||
|
||||
---
|
||||
|
||||
*Last updated: 2025-10-26.*
|
||||
# Internal-Only Policy Example (`internal-only.stella`)
|
||||
|
||||
A relaxed profile for internal services and development environments: allow Medium severities with warnings, rely on VEX more heavily, but still block KEV/actively exploited advisories.
|
||||
|
||||
```dsl
|
||||
policy "Internal Only Policy" syntax "stella-dsl@1" {
|
||||
metadata {
|
||||
description = "Lenient policy for internal / dev tenants."
|
||||
tags = ["internal","dev"]
|
||||
}
|
||||
|
||||
profile severity {
|
||||
env exposure_adjustments {
|
||||
if env.exposure == "internal" then -0.4
|
||||
if env.stage == "dev" then -0.6
|
||||
}
|
||||
}
|
||||
|
||||
rule block_kev priority 1 {
|
||||
when advisory.has_tag("kev")
|
||||
then status := "blocked"
|
||||
because "Known exploited vulnerabilities must be remediated."
|
||||
}
|
||||
|
||||
rule allow_medium_with_warning {
|
||||
when severity.normalized == "Medium"
|
||||
and env.exposure == "internal"
|
||||
then warn message "Medium severity permitted in internal environments."
|
||||
because "Allow Medium findings with warning for internal workloads."
|
||||
}
|
||||
|
||||
rule accept_vendor_vex {
|
||||
when vex.any(status in ["not_affected","fixed"])
|
||||
then status := vex.status
|
||||
annotate justification := vex.latest().justification
|
||||
because "Trust vendor VEX statements for internal scope."
|
||||
}
|
||||
|
||||
rule quiet_low_priority {
|
||||
when severity.normalized <= "Low"
|
||||
then ignore until "2026-01-01T00:00:00Z"
|
||||
because "Quiet low severity until next annual remediation sweep."
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Commentary
|
||||
|
||||
- Suitable for staging/dev tenants with lower blast radius.
|
||||
- KEV advisories override lenient behaviour to maintain minimum security bar.
|
||||
- Warnings ensure Medium findings stay visible in dashboards and CLI outputs.
|
||||
- Quiet rule enforces planned clean-up date; update before expiry.
|
||||
|
||||
## Try it out
|
||||
|
||||
```bash
|
||||
stella policy lint examples/policies/internal-only.stella
|
||||
stella policy simulate P-internal --candidate 1 \
|
||||
--sbom sbom:internal-service --env exposure=internal --env stage=dev
|
||||
```
|
||||
|
||||
## Compliance checklist
|
||||
|
||||
- [ ] Tenant classified as internal-only with documented risk acceptance.
|
||||
- [ ] KEV feed synced (Concelier) and tags confirmed before relying on rule.
|
||||
- [ ] Quiet expiry tracked; remediation backlog updated prior to deadline.
|
||||
- [ ] Developers informed that warnings still affect quality score.
|
||||
- [ ] Policy not used for production or internet-exposed services.
|
||||
|
||||
---
|
||||
|
||||
*Last updated: 2025-10-26.*
|
||||
|
||||
@@ -1,39 +1,39 @@
|
||||
policy "Internal Only Policy" syntax "stella-dsl@1" {
|
||||
metadata {
|
||||
description = "Lenient policy for internal / dev tenants."
|
||||
tags = ["internal","dev"]
|
||||
}
|
||||
|
||||
profile severity {
|
||||
env exposure_adjustments {
|
||||
if env.exposure == "internal" then -0.4
|
||||
if env.stage == "dev" then -0.6
|
||||
}
|
||||
}
|
||||
|
||||
rule block_kev priority 1 {
|
||||
when advisory.has_tag("kev")
|
||||
then status := "blocked"
|
||||
because "Known exploited vulnerabilities must be remediated."
|
||||
}
|
||||
|
||||
rule allow_medium_with_warning {
|
||||
when severity.normalized == "Medium"
|
||||
and env.exposure == "internal"
|
||||
then warn message "Medium severity permitted in internal environments."
|
||||
because "Allow Medium findings with warning for internal workloads."
|
||||
}
|
||||
|
||||
rule accept_vendor_vex {
|
||||
when vex.any(status in ["not_affected","fixed"])
|
||||
then status := vex.status
|
||||
annotate justification := vex.latest().justification
|
||||
because "Trust vendor VEX statements for internal scope."
|
||||
}
|
||||
|
||||
rule quiet_low_priority {
|
||||
when severity.normalized <= "Low"
|
||||
then ignore until "2026-01-01T00:00:00Z"
|
||||
because "Quiet low severity until next annual remediation sweep."
|
||||
}
|
||||
}
|
||||
policy "Internal Only Policy" syntax "stella-dsl@1" {
|
||||
metadata {
|
||||
description = "Lenient policy for internal / dev tenants."
|
||||
tags = ["internal","dev"]
|
||||
}
|
||||
|
||||
profile severity {
|
||||
env exposure_adjustments {
|
||||
if env.exposure == "internal" then -0.4
|
||||
if env.stage == "dev" then -0.6
|
||||
}
|
||||
}
|
||||
|
||||
rule block_kev priority 1 {
|
||||
when advisory.has_tag("kev")
|
||||
then status := "blocked"
|
||||
because "Known exploited vulnerabilities must be remediated."
|
||||
}
|
||||
|
||||
rule allow_medium_with_warning {
|
||||
when severity.normalized == "Medium"
|
||||
and env.exposure == "internal"
|
||||
then warn message "Medium severity permitted in internal environments."
|
||||
because "Allow Medium findings with warning for internal workloads."
|
||||
}
|
||||
|
||||
rule accept_vendor_vex {
|
||||
when vex.any(status in ["not_affected","fixed"])
|
||||
then status := vex.status
|
||||
annotate justification := vex.latest().justification
|
||||
because "Trust vendor VEX statements for internal scope."
|
||||
}
|
||||
|
||||
rule quiet_low_priority {
|
||||
when severity.normalized <= "Low"
|
||||
then ignore until "2026-01-01T00:00:00Z"
|
||||
because "Quiet low severity until next annual remediation sweep."
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1,31 +1,31 @@
|
||||
version: "1.0"
|
||||
metadata:
|
||||
description: Relaxed internal/development policy
|
||||
tags:
|
||||
- internal
|
||||
- dev
|
||||
rules:
|
||||
- name: Block KEV advisories
|
||||
tags: [kev]
|
||||
action: block
|
||||
|
||||
- name: Warn medium severity
|
||||
severity: [Medium]
|
||||
environments: [internal]
|
||||
action: warn
|
||||
|
||||
- name: Accept vendor VEX
|
||||
action:
|
||||
type: require_vex
|
||||
requireVex:
|
||||
vendors: [VendorX, VendorY]
|
||||
justifications:
|
||||
- component_not_present
|
||||
- vulnerable_code_not_present
|
||||
|
||||
- name: Quiet low severity
|
||||
severity: [Low, Informational]
|
||||
action:
|
||||
type: ignore
|
||||
until: 2026-01-01T00:00:00Z
|
||||
justification: "Deferred to annual remediation cycle"
|
||||
version: "1.0"
|
||||
metadata:
|
||||
description: Relaxed internal/development policy
|
||||
tags:
|
||||
- internal
|
||||
- dev
|
||||
rules:
|
||||
- name: Block KEV advisories
|
||||
tags: [kev]
|
||||
action: block
|
||||
|
||||
- name: Warn medium severity
|
||||
severity: [Medium]
|
||||
environments: [internal]
|
||||
action: warn
|
||||
|
||||
- name: Accept vendor VEX
|
||||
action:
|
||||
type: require_vex
|
||||
requireVex:
|
||||
vendors: [VendorX, VendorY]
|
||||
justifications:
|
||||
- component_not_present
|
||||
- vulnerable_code_not_present
|
||||
|
||||
- name: Quiet low severity
|
||||
severity: [Low, Informational]
|
||||
action:
|
||||
type: ignore
|
||||
until: 2026-01-01T00:00:00Z
|
||||
justification: "Deferred to annual remediation cycle"
|
||||
|
||||
@@ -1,72 +1,72 @@
|
||||
# Serverless Policy Example (`serverless.stella`)
|
||||
|
||||
Optimised for short-lived serverless workloads: focus on runtime integrity, disallow vulnerable layers entirely, and permit temporary suppressions only with strict justification windows.
|
||||
|
||||
```dsl
|
||||
policy "Serverless Tight Policy" syntax "stella-dsl@1" {
|
||||
metadata {
|
||||
description = "Aggressive blocking for serverless runtimes."
|
||||
tags = ["serverless","prod","strict"]
|
||||
}
|
||||
|
||||
profile severity {
|
||||
env runtime_overrides {
|
||||
if env.runtime == "serverless" then +0.7
|
||||
if env.runtime == "batch" then +0.2
|
||||
}
|
||||
}
|
||||
|
||||
rule block_any_high {
|
||||
when severity.normalized >= "High"
|
||||
then status := "blocked"
|
||||
because "Serverless workloads block High+ severities."
|
||||
}
|
||||
|
||||
rule forbid_unpinned_base {
|
||||
when sbom.has_tag("image:latest-tag")
|
||||
then status := "blocked"
|
||||
because "Base image must be pinned (no :latest)."
|
||||
}
|
||||
|
||||
rule zero_tolerance_vex {
|
||||
when vex.any(status == "not_affected")
|
||||
then requireVex { vendors = ["VendorX","VendorY"], justifications = ["component_not_present"] }
|
||||
because "Allow not_affected only from trusted vendors with strongest justification."
|
||||
}
|
||||
|
||||
rule temporary_quiet {
|
||||
when env.deployment == "canary"
|
||||
and severity.normalized == "Medium"
|
||||
then ignore until coalesce(env.quietUntil, "2025-12-31T00:00:00Z")
|
||||
because "Allow short canary quiet window while fix rolls out."
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Commentary
|
||||
|
||||
- Designed for serverless tenants where redeploy cost is low and failing fast is preferred.
|
||||
- `forbid_unpinned_base` enforces supply-chain best practices.
|
||||
- `temporary_quiet` ensures quiet windows expire automatically; require deployments to set `env.quietUntil`.
|
||||
- Intended to be layered on top of baseline (override per tenant) or used standalone for serverless-only accounts.
|
||||
|
||||
## Try it out
|
||||
|
||||
```bash
|
||||
stella policy lint examples/policies/serverless.stella
|
||||
stella policy simulate P-serverless --candidate 1 \
|
||||
--sbom sbom:lambda-hello --env runtime=serverless --env deployment=canary
|
||||
```
|
||||
|
||||
## Compliance checklist
|
||||
|
||||
- [ ] Quiet window expirations tracked and documented.
|
||||
- [ ] Trusted VEX vendor list reviewed quarterly.
|
||||
- [ ] Deployment pipeline enforces pinned base images before approval.
|
||||
- [ ] Canary deployments monitored for recurrence before ignoring Medium severity.
|
||||
- [ ] Serverless teams acknowledge runbook for blocked deployments.
|
||||
|
||||
---
|
||||
|
||||
*Last updated: 2025-10-26.*
|
||||
|
||||
# Serverless Policy Example (`serverless.stella`)
|
||||
|
||||
Optimised for short-lived serverless workloads: focus on runtime integrity, disallow vulnerable layers entirely, and permit temporary suppressions only with strict justification windows.
|
||||
|
||||
```dsl
|
||||
policy "Serverless Tight Policy" syntax "stella-dsl@1" {
|
||||
metadata {
|
||||
description = "Aggressive blocking for serverless runtimes."
|
||||
tags = ["serverless","prod","strict"]
|
||||
}
|
||||
|
||||
profile severity {
|
||||
env runtime_overrides {
|
||||
if env.runtime == "serverless" then +0.7
|
||||
if env.runtime == "batch" then +0.2
|
||||
}
|
||||
}
|
||||
|
||||
rule block_any_high {
|
||||
when severity.normalized >= "High"
|
||||
then status := "blocked"
|
||||
because "Serverless workloads block High+ severities."
|
||||
}
|
||||
|
||||
rule forbid_unpinned_base {
|
||||
when sbom.has_tag("image:latest-tag")
|
||||
then status := "blocked"
|
||||
because "Base image must be pinned (no :latest)."
|
||||
}
|
||||
|
||||
rule zero_tolerance_vex {
|
||||
when vex.any(status == "not_affected")
|
||||
then requireVex { vendors = ["VendorX","VendorY"], justifications = ["component_not_present"] }
|
||||
because "Allow not_affected only from trusted vendors with strongest justification."
|
||||
}
|
||||
|
||||
rule temporary_quiet {
|
||||
when env.deployment == "canary"
|
||||
and severity.normalized == "Medium"
|
||||
then ignore until coalesce(env.quietUntil, "2025-12-31T00:00:00Z")
|
||||
because "Allow short canary quiet window while fix rolls out."
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Commentary
|
||||
|
||||
- Designed for serverless tenants where redeploy cost is low and failing fast is preferred.
|
||||
- `forbid_unpinned_base` enforces supply-chain best practices.
|
||||
- `temporary_quiet` ensures quiet windows expire automatically; require deployments to set `env.quietUntil`.
|
||||
- Intended to be layered on top of baseline (override per tenant) or used standalone for serverless-only accounts.
|
||||
|
||||
## Try it out
|
||||
|
||||
```bash
|
||||
stella policy lint examples/policies/serverless.stella
|
||||
stella policy simulate P-serverless --candidate 1 \
|
||||
--sbom sbom:lambda-hello --env runtime=serverless --env deployment=canary
|
||||
```
|
||||
|
||||
## Compliance checklist
|
||||
|
||||
- [ ] Quiet window expirations tracked and documented.
|
||||
- [ ] Trusted VEX vendor list reviewed quarterly.
|
||||
- [ ] Deployment pipeline enforces pinned base images before approval.
|
||||
- [ ] Canary deployments monitored for recurrence before ignoring Medium severity.
|
||||
- [ ] Serverless teams acknowledge runbook for blocked deployments.
|
||||
|
||||
---
|
||||
|
||||
*Last updated: 2025-10-26.*
|
||||
|
||||
|
||||
@@ -1,39 +1,39 @@
|
||||
policy "Serverless Tight Policy" syntax "stella-dsl@1" {
|
||||
metadata {
|
||||
description = "Aggressive blocking for serverless runtimes."
|
||||
tags = ["serverless","prod","strict"]
|
||||
}
|
||||
|
||||
profile severity {
|
||||
env runtime_overrides {
|
||||
if env.runtime == "serverless" then +0.7
|
||||
if env.runtime == "batch" then +0.2
|
||||
}
|
||||
}
|
||||
|
||||
rule block_any_high {
|
||||
when severity.normalized >= "High"
|
||||
then status := "blocked"
|
||||
because "Serverless workloads block High+ severities."
|
||||
}
|
||||
|
||||
rule forbid_unpinned_base {
|
||||
when sbom.has_tag("image:latest-tag")
|
||||
then status := "blocked"
|
||||
because "Base image must be pinned (no :latest)."
|
||||
}
|
||||
|
||||
rule zero_tolerance_vex {
|
||||
when vex.any(status == "not_affected")
|
||||
then requireVex { vendors = ["VendorX","VendorY"], justifications = ["component_not_present"] }
|
||||
because "Allow not_affected only from trusted vendors with strongest justification."
|
||||
}
|
||||
|
||||
rule temporary_quiet {
|
||||
when env.deployment == "canary"
|
||||
and severity.normalized == "Medium"
|
||||
then ignore until coalesce(env.quietUntil, "2025-12-31T00:00:00Z")
|
||||
because "Allow short canary quiet window while fix rolls out."
|
||||
}
|
||||
}
|
||||
|
||||
policy "Serverless Tight Policy" syntax "stella-dsl@1" {
|
||||
metadata {
|
||||
description = "Aggressive blocking for serverless runtimes."
|
||||
tags = ["serverless","prod","strict"]
|
||||
}
|
||||
|
||||
profile severity {
|
||||
env runtime_overrides {
|
||||
if env.runtime == "serverless" then +0.7
|
||||
if env.runtime == "batch" then +0.2
|
||||
}
|
||||
}
|
||||
|
||||
rule block_any_high {
|
||||
when severity.normalized >= "High"
|
||||
then status := "blocked"
|
||||
because "Serverless workloads block High+ severities."
|
||||
}
|
||||
|
||||
rule forbid_unpinned_base {
|
||||
when sbom.has_tag("image:latest-tag")
|
||||
then status := "blocked"
|
||||
because "Base image must be pinned (no :latest)."
|
||||
}
|
||||
|
||||
rule zero_tolerance_vex {
|
||||
when vex.any(status == "not_affected")
|
||||
then requireVex { vendors = ["VendorX","VendorY"], justifications = ["component_not_present"] }
|
||||
because "Allow not_affected only from trusted vendors with strongest justification."
|
||||
}
|
||||
|
||||
rule temporary_quiet {
|
||||
when env.deployment == "canary"
|
||||
and severity.normalized == "Medium"
|
||||
then ignore until coalesce(env.quietUntil, "2025-12-31T00:00:00Z")
|
||||
because "Allow short canary quiet window while fix rolls out."
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -1,41 +1,41 @@
|
||||
version: "1.0"
|
||||
metadata:
|
||||
description: Strict policy for serverless workloads
|
||||
tags:
|
||||
- serverless
|
||||
- prod
|
||||
- strict
|
||||
exceptions:
|
||||
effects:
|
||||
- id: suppress-canary
|
||||
name: Canary Freeze
|
||||
effect: suppress
|
||||
routingTemplate: secops-approvers
|
||||
maxDurationDays: 14
|
||||
routingTemplates:
|
||||
- id: secops-approvers
|
||||
authorityRouteId: governance.secops
|
||||
requireMfa: true
|
||||
rules:
|
||||
- name: Block High And Above
|
||||
severity: [High, Critical]
|
||||
action: block
|
||||
|
||||
- name: Forbid Unpinned Base Images
|
||||
tags: [image:latest-tag]
|
||||
action: block
|
||||
|
||||
- name: Require Trusted VEX
|
||||
action:
|
||||
type: require_vex
|
||||
requireVex:
|
||||
vendors: [VendorX, VendorY]
|
||||
justifications: [component_not_present]
|
||||
|
||||
- name: Quiet Medium Canary
|
||||
severity: [Medium]
|
||||
environments: [canary]
|
||||
action:
|
||||
type: ignore
|
||||
until: 2025-12-31T00:00:00Z
|
||||
justification: "Temporary canary exception"
|
||||
version: "1.0"
|
||||
metadata:
|
||||
description: Strict policy for serverless workloads
|
||||
tags:
|
||||
- serverless
|
||||
- prod
|
||||
- strict
|
||||
exceptions:
|
||||
effects:
|
||||
- id: suppress-canary
|
||||
name: Canary Freeze
|
||||
effect: suppress
|
||||
routingTemplate: secops-approvers
|
||||
maxDurationDays: 14
|
||||
routingTemplates:
|
||||
- id: secops-approvers
|
||||
authorityRouteId: governance.secops
|
||||
requireMfa: true
|
||||
rules:
|
||||
- name: Block High And Above
|
||||
severity: [High, Critical]
|
||||
action: block
|
||||
|
||||
- name: Forbid Unpinned Base Images
|
||||
tags: [image:latest-tag]
|
||||
action: block
|
||||
|
||||
- name: Require Trusted VEX
|
||||
action:
|
||||
type: require_vex
|
||||
requireVex:
|
||||
vendors: [VendorX, VendorY]
|
||||
justifications: [component_not_present]
|
||||
|
||||
- name: Quiet Medium Canary
|
||||
severity: [Medium]
|
||||
environments: [canary]
|
||||
action:
|
||||
type: ignore
|
||||
until: 2025-12-31T00:00:00Z
|
||||
justification: "Temporary canary exception"
|
||||
|
||||
@@ -1,154 +1,154 @@
|
||||
# StellaOps Console – Guided Tours (Sprint 23)
|
||||
|
||||
> **Audience:** Field enablement, Docs Guild writers, Console product leads, and onboarding facilitators.
|
||||
> **Scope:** Ready-to-run walkthrough scripts that showcase the Console’s critical workflows—triage, audit evidence, and policy rollout—while reinforcing CLI parity, tenancy, and offline expectations.
|
||||
|
||||
These tours stitch together the primary Console workspaces so trainers can deliver consistent demos or capture annotated media (screenshots/GIFs). Each tour lists prerequisites, live steps, CLI fallbacks, and assets to capture. Use them alongside the workspace dossiers in `/docs/ui/*.md` when preparing customer sessions or internal dry runs.
|
||||
|
||||
---
|
||||
|
||||
## 1 · Prerequisites & Setup
|
||||
|
||||
- **Environment:** Console deployed per [deployment guide](../deploy/console.md) with Scheduler, Policy Engine, Concelier, Excititor, SBOM Service, and Downloads manifest available.
|
||||
- **Tenant & data:** Sample tenant populated with recent scans, findings, runs, and export bundles. Ensure Offline Kit snapshot exists for offline callouts.
|
||||
- **Scopes:** Presenter identity must hold `ui.read`, `findings.read`, `policy:*` (read/write/simulate/approve), `runs.read`, `downloads.read`, `aoc:verify`, and `ui.telemetry` to surface telemetry banners.
|
||||
- **Browser tooling:** Enable screen recording (1920×1080 @ 60 fps) and keyboard overlay if capturing walkthroughs.
|
||||
- **CLI parity:** Have `stella` CLI configured against the same tenant; keep terminal window ready for parity steps.
|
||||
- **Assets directory:** Store captures under `docs/assets/ui/tours/` (see [`README`](../assets/ui/tours/README.md)) with the naming convention `<tour>-step-<nn>.png` and `<tour>-flow.gif`.
|
||||
|
||||
---
|
||||
|
||||
## 2 · Tour A — Critical Finding Triage
|
||||
|
||||
**Persona:** Security analyst responding to a fresh high-severity finding.
|
||||
**Goal:** Navigate from dashboard signal to remediation decision, highlighting explain trails and run evidence.
|
||||
|
||||
### 2.1 Key references
|
||||
- [Console overview](../ui/console-overview.md) – tenant switching, status ticker.
|
||||
- [Navigation](../ui/navigation.md) – command palette, shortcuts.
|
||||
- [Findings workspace](../ui/findings.md) – filters, explain drawer, exports.
|
||||
- [Runs workspace](../ui/runs.md) – live progress, evidence downloads.
|
||||
|
||||
### 2.2 Live walkthrough
|
||||
1. **Start on Dashboard:** Show status ticker surfacing new `Critical` badge. Call out tenant pill and offline banner behaviour (§3 of console overview).
|
||||
2. **Command palette jump:** Press `Ctrl/Cmd+K`, type `Findings`, hit `Enter`. Narrate keyboard accessibility from navigation guide.
|
||||
3. **Apply global filters:** Open filter tray (`Shift+F`), set `Severity = Critical`, `Status = affected`, time window `Last 24h`. Mention saved view presets triggered with `Ctrl/Cmd+1`.
|
||||
4. **Open explain drawer:** Select top finding, trigger `Explain` tab. Highlight rule chain, VEX impact, and evidence references (§5 of findings doc).
|
||||
5. **Dive into related run:** Click `Run ID` link inside explain drawer → opens Runs detail drawer filtered by run ID. Show segmented progress SSE updates.
|
||||
6. **Capture evidence:** In Runs drawer, download evidence bundle; note CLI parity `stella runs export --run <id>`. Mention offline fallback (download queue offline banner from runs doc §10).
|
||||
7. **Escalate / create ticket:** Use bulk action or comment (if configured) to demonstrate optional integration; mention Authority audit log tie-in.
|
||||
8. **Wrap with CLI:** Pop terminal and run `stella findings explain --policy <id> --finding <key> --format markdown` to show reproducibility.
|
||||
|
||||
### 2.3 Capture checklist
|
||||
- `docs/assets/ui/tours/triage-step-01.png` — dashboard ticker highlighting new criticals.
|
||||
|
||||
- `docs/assets/ui/tours/triage-step-03.png` — filter tray with severity/time window applied.
|
||||
|
||||
- `docs/assets/ui/tours/triage-step-04.png` — explain drawer evidence tab.
|
||||
|
||||
- `docs/assets/ui/tours/triage-flow.gif` — 20 s screen recording of steps 1–5 with annotations.
|
||||
|
||||
|
||||
### 2.4 Talking points & callouts
|
||||
- Call out Aggregation-Only boundaries: findings reference Concelier/Excititor provenance, UI stays read-only.
|
||||
- Mention `ui_route_render_seconds` telemetry for demos (see [observability guide](../observability/ui-telemetry.md)).
|
||||
- Offline note: highlight offline banner that appears if `/console/status` heartbeat fails (§6 of console overview).
|
||||
|
||||
---
|
||||
|
||||
## 3 · Tour B — Audit Evidence Export
|
||||
|
||||
**Persona:** Compliance lead compiling artefacts for an external audit.
|
||||
**Goal:** Retrieve signed manifests, export run/finding evidence, and verify parity with Offline Kit.
|
||||
|
||||
### 3.1 Key references
|
||||
- [Downloads workspace](../ui/downloads.md) – manifest, parity, export queue.
|
||||
- [Runs workspace](../ui/runs.md) – evidence panel.
|
||||
- [Console security posture](../security/console-security.md) – evidence handling.
|
||||
- [CLI vs UI parity matrix](../cli-vs-ui-parity.md).
|
||||
|
||||
### 3.2 Live walkthrough
|
||||
1. **Open Downloads:** Use left rail or command palette to reach `/console/downloads`. Point out snapshot banner, cosign verification status.
|
||||
2. **Verify manifest:** Click “Verify signature” quick action; narrate parity with `cosign verify --key <key> manifest.json` from downloads doc §3.
|
||||
3. **Compare Offline Kit:** Switch to “Offline Kits” tab, run parity check to ensure kit digest matches manifest. Demonstrate offline guidance (downloads doc §6).
|
||||
4. **Queue evidence bundle:** Navigate to Runs workspace, choose relevant run, trigger “Bundle for offline” (runs doc §8).
|
||||
5. **Return to Downloads → Exports tab:** Show newly generated evidence bundle with retention countdown.
|
||||
6. **Download & inspect:** Open detail drawer, copy CLI command `stella runs export --run <id> --bundle`. Mention location for storing evidence.
|
||||
7. **Log parity results:** Use notes or tags to flag audit package completion (if notifications configured).
|
||||
8. **CLI parity close-out:** Run `stella downloads manifest --channel stable` to mirror UI manifest retrieval. Confirm digests match.
|
||||
|
||||
### 3.3 Capture checklist
|
||||
- `docs/assets/ui/tours/audit-step-02.png` — manifest verification banner (green).
|
||||
|
||||
- `docs/assets/ui/tours/audit-step-05.png` — exports tab showing evidence bundle ready.
|
||||
|
||||
- `docs/assets/ui/tours/audit-flow.gif` — 25 s capture from manifest view through export download.
|
||||
|
||||
|
||||
### 3.4 Talking points & callouts
|
||||
- Stress deterministic manifests and Cosign signatures; reference deployment doc for TLS/CSP alignment.
|
||||
- Highlight audit trail: downloads actions recorded via `ui.download.commandCopied` logs and Authority audit entries.
|
||||
- Offline note: show guidance when parity check detects stale manifest; mention CLI fallback for sealed networks.
|
||||
|
||||
---
|
||||
|
||||
## 4 · Tour C — Policy Rollout & Promotion
|
||||
|
||||
**Persona:** Policy owner preparing and promoting a new ruleset.
|
||||
**Goal:** Draft review, simulation, approval, and promotion within Console, with CLI parity.
|
||||
|
||||
### 4.1 Key references
|
||||
- [Policies workspace](../ui/policies.md) – simulations, approvals, promotion.
|
||||
- [Policy editor](../ui/policy-editor.md) – Monaco editor, linting.
|
||||
- [Runs workspace](../ui/runs.md) – policy run monitoring.
|
||||
- [Security posture](../security/console-security.md) – fresh-auth and scopes.
|
||||
|
||||
### 4.2 Live walkthrough
|
||||
1. **Policy overview:** Open `/console/policies`, filter by “Staged” state. Highlight list columns (owners, pending approvals).
|
||||
2. **Enter draft:** Select policy → open editor view. Show checklist sidebar (lint, simulation, determinism).
|
||||
3. **Run lint & simulation:** Hit `Run lint`, then `Run simulation`. Narrate asynchronous progress with SSE ticker; reference CLI `stella policy simulate`.
|
||||
4. **Review diff:** Open simulation diff view to compare Active vs Staged; highlight severity up/down badges (§6 of policies doc).
|
||||
5. **Approval workflow:** Assign reviewer, show comment thread. Trigger fresh-auth prompt when clicking “Submit for review” (security doc §1.2).
|
||||
6. **Promote policy:** After approvals, open promotion dialog, choose “Full run”. Emphasise policy run scheduling and RBAC.
|
||||
7. **Monitor run:** Jump to Runs workspace, filter by policy run; show progress segments and findings delta metrics.
|
||||
8. **Publish CLI parity:** Execute `stella policy promote --policy <id> --revision <rev> --run-mode full` to reinforce reproducibility.
|
||||
|
||||
### 4.3 Capture checklist
|
||||
- `docs/assets/ui/tours/policy-step-02.png` — editor checklist with lint/simulation statuses.
|
||||
|
||||
- `docs/assets/ui/tours/policy-step-04.png` — simulation diff comparing Active vs Staged.
|
||||
|
||||
- `docs/assets/ui/tours/policy-flow.gif` — 30 s clip from draft view through promotion confirmation.
|
||||
|
||||
|
||||
### 4.4 Talking points & callouts
|
||||
- Stress governance: approvals logged with correlation IDs, fresh-auth enforced.
|
||||
- Mention telemetry metrics (`ui_tenant_switch_total`, policy run charts) for monitoring adoption.
|
||||
- Offline note: show how promotion dialog surfaces CLI script when in sealed mode; reference offline guidance in policies doc §10.
|
||||
|
||||
---
|
||||
|
||||
## 5 · Production Tips & Media Hygiene
|
||||
|
||||
- **Script timing:** Keep each tour ≤ 3 minutes live demo, ≤ 30 s GIF. Include captions for accessibility.
|
||||
- **Annotations:** Use consistent callouts (numbered badges, short labels) overlayed in post-processing; ensure final media compressed but legible (< 2 MB PNG, < 8 MB GIF). See `docs/assets/ui/tours/README.md` for shared template guidance.
|
||||
- **Versioning:** Annotated assets should include Console build hash in metadata or caption (align with `/console/downloads` manifest version).
|
||||
- **Storage:** Commit final media under `docs/assets/ui/tours/` and update `.gitattributes` if smudge filters required. Note large GIFs may need Git LFS depending on repository policy.
|
||||
- **Review cadence:** Re-run tours whenever workspaces change navigation or introduce new buttons; log updates in `docs/updates/<date>-console-tours.md` (create if absent).
|
||||
|
||||
---
|
||||
|
||||
## 6 · Compliance Checklist
|
||||
|
||||
- [x] Tour scripts cover triage, audit evidence, and policy rollout scenarios requested in DOCS-CONSOLE-23-017.
|
||||
- [x] Each tour references authoritative workspace docs and CLI parity commands.
|
||||
- [x] Capture checklist names align with `docs/assets/ui/tours/` convention.
|
||||
- [x] Offline and sealed-mode notes included for every flow.
|
||||
- [x] Security considerations (scopes, fresh-auth, evidence handling) highlighted.
|
||||
- [x] Observability/telemetry pointers surfaced to support Ops follow-up.
|
||||
- [x] Media hygiene guidance documented (assets, compression, versioning).
|
||||
- [x] Document timestamp reflects Sprint 23 delivery.
|
||||
|
||||
---
|
||||
|
||||
*Last updated: 2025-10-27 (Sprint 23).*
|
||||
# StellaOps Console – Guided Tours (Sprint 23)
|
||||
|
||||
> **Audience:** Field enablement, Docs Guild writers, Console product leads, and onboarding facilitators.
|
||||
> **Scope:** Ready-to-run walkthrough scripts that showcase the Console’s critical workflows—triage, audit evidence, and policy rollout—while reinforcing CLI parity, tenancy, and offline expectations.
|
||||
|
||||
These tours stitch together the primary Console workspaces so trainers can deliver consistent demos or capture annotated media (screenshots/GIFs). Each tour lists prerequisites, live steps, CLI fallbacks, and assets to capture. Use them alongside the workspace dossiers in `/docs/ui/*.md` when preparing customer sessions or internal dry runs.
|
||||
|
||||
---
|
||||
|
||||
## 1 · Prerequisites & Setup
|
||||
|
||||
- **Environment:** Console deployed per [deployment guide](../deploy/console.md) with Scheduler, Policy Engine, Concelier, Excititor, SBOM Service, and Downloads manifest available.
|
||||
- **Tenant & data:** Sample tenant populated with recent scans, findings, runs, and export bundles. Ensure Offline Kit snapshot exists for offline callouts.
|
||||
- **Scopes:** Presenter identity must hold `ui.read`, `findings.read`, `policy:*` (read/write/simulate/approve), `runs.read`, `downloads.read`, `aoc:verify`, and `ui.telemetry` to surface telemetry banners.
|
||||
- **Browser tooling:** Enable screen recording (1920×1080 @ 60 fps) and keyboard overlay if capturing walkthroughs.
|
||||
- **CLI parity:** Have `stella` CLI configured against the same tenant; keep terminal window ready for parity steps.
|
||||
- **Assets directory:** Store captures under `docs/assets/ui/tours/` (see [`README`](../assets/ui/tours/README.md)) with the naming convention `<tour>-step-<nn>.png` and `<tour>-flow.gif`.
|
||||
|
||||
---
|
||||
|
||||
## 2 · Tour A — Critical Finding Triage
|
||||
|
||||
**Persona:** Security analyst responding to a fresh high-severity finding.
|
||||
**Goal:** Navigate from dashboard signal to remediation decision, highlighting explain trails and run evidence.
|
||||
|
||||
### 2.1 Key references
|
||||
- [Console overview](../ui/console-overview.md) – tenant switching, status ticker.
|
||||
- [Navigation](../ui/navigation.md) – command palette, shortcuts.
|
||||
- [Findings workspace](../ui/findings.md) – filters, explain drawer, exports.
|
||||
- [Runs workspace](../ui/runs.md) – live progress, evidence downloads.
|
||||
|
||||
### 2.2 Live walkthrough
|
||||
1. **Start on Dashboard:** Show status ticker surfacing new `Critical` badge. Call out tenant pill and offline banner behaviour (§3 of console overview).
|
||||
2. **Command palette jump:** Press `Ctrl/Cmd+K`, type `Findings`, hit `Enter`. Narrate keyboard accessibility from navigation guide.
|
||||
3. **Apply global filters:** Open filter tray (`Shift+F`), set `Severity = Critical`, `Status = affected`, time window `Last 24h`. Mention saved view presets triggered with `Ctrl/Cmd+1`.
|
||||
4. **Open explain drawer:** Select top finding, trigger `Explain` tab. Highlight rule chain, VEX impact, and evidence references (§5 of findings doc).
|
||||
5. **Dive into related run:** Click `Run ID` link inside explain drawer → opens Runs detail drawer filtered by run ID. Show segmented progress SSE updates.
|
||||
6. **Capture evidence:** In Runs drawer, download evidence bundle; note CLI parity `stella runs export --run <id>`. Mention offline fallback (download queue offline banner from runs doc §10).
|
||||
7. **Escalate / create ticket:** Use bulk action or comment (if configured) to demonstrate optional integration; mention Authority audit log tie-in.
|
||||
8. **Wrap with CLI:** Pop terminal and run `stella findings explain --policy <id> --finding <key> --format markdown` to show reproducibility.
|
||||
|
||||
### 2.3 Capture checklist
|
||||
- `docs/assets/ui/tours/triage-step-01.png` — dashboard ticker highlighting new criticals.
|
||||
|
||||
- `docs/assets/ui/tours/triage-step-03.png` — filter tray with severity/time window applied.
|
||||
|
||||
- `docs/assets/ui/tours/triage-step-04.png` — explain drawer evidence tab.
|
||||
|
||||
- `docs/assets/ui/tours/triage-flow.gif` — 20 s screen recording of steps 1–5 with annotations.
|
||||
|
||||
|
||||
### 2.4 Talking points & callouts
|
||||
- Call out Aggregation-Only boundaries: findings reference Concelier/Excititor provenance, UI stays read-only.
|
||||
- Mention `ui_route_render_seconds` telemetry for demos (see [observability guide](../observability/ui-telemetry.md)).
|
||||
- Offline note: highlight offline banner that appears if `/console/status` heartbeat fails (§6 of console overview).
|
||||
|
||||
---
|
||||
|
||||
## 3 · Tour B — Audit Evidence Export
|
||||
|
||||
**Persona:** Compliance lead compiling artefacts for an external audit.
|
||||
**Goal:** Retrieve signed manifests, export run/finding evidence, and verify parity with Offline Kit.
|
||||
|
||||
### 3.1 Key references
|
||||
- [Downloads workspace](../ui/downloads.md) – manifest, parity, export queue.
|
||||
- [Runs workspace](../ui/runs.md) – evidence panel.
|
||||
- [Console security posture](../security/console-security.md) – evidence handling.
|
||||
- [CLI vs UI parity matrix](../cli-vs-ui-parity.md).
|
||||
|
||||
### 3.2 Live walkthrough
|
||||
1. **Open Downloads:** Use left rail or command palette to reach `/console/downloads`. Point out snapshot banner, cosign verification status.
|
||||
2. **Verify manifest:** Click “Verify signature” quick action; narrate parity with `cosign verify --key <key> manifest.json` from downloads doc §3.
|
||||
3. **Compare Offline Kit:** Switch to “Offline Kits” tab, run parity check to ensure kit digest matches manifest. Demonstrate offline guidance (downloads doc §6).
|
||||
4. **Queue evidence bundle:** Navigate to Runs workspace, choose relevant run, trigger “Bundle for offline” (runs doc §8).
|
||||
5. **Return to Downloads → Exports tab:** Show newly generated evidence bundle with retention countdown.
|
||||
6. **Download & inspect:** Open detail drawer, copy CLI command `stella runs export --run <id> --bundle`. Mention location for storing evidence.
|
||||
7. **Log parity results:** Use notes or tags to flag audit package completion (if notifications configured).
|
||||
8. **CLI parity close-out:** Run `stella downloads manifest --channel stable` to mirror UI manifest retrieval. Confirm digests match.
|
||||
|
||||
### 3.3 Capture checklist
|
||||
- `docs/assets/ui/tours/audit-step-02.png` — manifest verification banner (green).
|
||||
|
||||
- `docs/assets/ui/tours/audit-step-05.png` — exports tab showing evidence bundle ready.
|
||||
|
||||
- `docs/assets/ui/tours/audit-flow.gif` — 25 s capture from manifest view through export download.
|
||||
|
||||
|
||||
### 3.4 Talking points & callouts
|
||||
- Stress deterministic manifests and Cosign signatures; reference deployment doc for TLS/CSP alignment.
|
||||
- Highlight audit trail: downloads actions recorded via `ui.download.commandCopied` logs and Authority audit entries.
|
||||
- Offline note: show guidance when parity check detects stale manifest; mention CLI fallback for sealed networks.
|
||||
|
||||
---
|
||||
|
||||
## 4 · Tour C — Policy Rollout & Promotion
|
||||
|
||||
**Persona:** Policy owner preparing and promoting a new ruleset.
|
||||
**Goal:** Draft review, simulation, approval, and promotion within Console, with CLI parity.
|
||||
|
||||
### 4.1 Key references
|
||||
- [Policies workspace](../ui/policies.md) – simulations, approvals, promotion.
|
||||
- [Policy editor](../ui/policy-editor.md) – Monaco editor, linting.
|
||||
- [Runs workspace](../ui/runs.md) – policy run monitoring.
|
||||
- [Security posture](../security/console-security.md) – fresh-auth and scopes.
|
||||
|
||||
### 4.2 Live walkthrough
|
||||
1. **Policy overview:** Open `/console/policies`, filter by “Staged” state. Highlight list columns (owners, pending approvals).
|
||||
2. **Enter draft:** Select policy → open editor view. Show checklist sidebar (lint, simulation, determinism).
|
||||
3. **Run lint & simulation:** Hit `Run lint`, then `Run simulation`. Narrate asynchronous progress with SSE ticker; reference CLI `stella policy simulate`.
|
||||
4. **Review diff:** Open simulation diff view to compare Active vs Staged; highlight severity up/down badges (§6 of policies doc).
|
||||
5. **Approval workflow:** Assign reviewer, show comment thread. Trigger fresh-auth prompt when clicking “Submit for review” (security doc §1.2).
|
||||
6. **Promote policy:** After approvals, open promotion dialog, choose “Full run”. Emphasise policy run scheduling and RBAC.
|
||||
7. **Monitor run:** Jump to Runs workspace, filter by policy run; show progress segments and findings delta metrics.
|
||||
8. **Publish CLI parity:** Execute `stella policy promote --policy <id> --revision <rev> --run-mode full` to reinforce reproducibility.
|
||||
|
||||
### 4.3 Capture checklist
|
||||
- `docs/assets/ui/tours/policy-step-02.png` — editor checklist with lint/simulation statuses.
|
||||
|
||||
- `docs/assets/ui/tours/policy-step-04.png` — simulation diff comparing Active vs Staged.
|
||||
|
||||
- `docs/assets/ui/tours/policy-flow.gif` — 30 s clip from draft view through promotion confirmation.
|
||||
|
||||
|
||||
### 4.4 Talking points & callouts
|
||||
- Stress governance: approvals logged with correlation IDs, fresh-auth enforced.
|
||||
- Mention telemetry metrics (`ui_tenant_switch_total`, policy run charts) for monitoring adoption.
|
||||
- Offline note: show how promotion dialog surfaces CLI script when in sealed mode; reference offline guidance in policies doc §10.
|
||||
|
||||
---
|
||||
|
||||
## 5 · Production Tips & Media Hygiene
|
||||
|
||||
- **Script timing:** Keep each tour ≤ 3 minutes live demo, ≤ 30 s GIF. Include captions for accessibility.
|
||||
- **Annotations:** Use consistent callouts (numbered badges, short labels) overlayed in post-processing; ensure final media compressed but legible (< 2 MB PNG, < 8 MB GIF). See `docs/assets/ui/tours/README.md` for shared template guidance.
|
||||
- **Versioning:** Annotated assets should include Console build hash in metadata or caption (align with `/console/downloads` manifest version).
|
||||
- **Storage:** Commit final media under `docs/assets/ui/tours/` and update `.gitattributes` if smudge filters required. Note large GIFs may need Git LFS depending on repository policy.
|
||||
- **Review cadence:** Re-run tours whenever workspaces change navigation or introduce new buttons; log updates in `docs/updates/<date>-console-tours.md` (create if absent).
|
||||
|
||||
---
|
||||
|
||||
## 6 · Compliance Checklist
|
||||
|
||||
- [x] Tour scripts cover triage, audit evidence, and policy rollout scenarios requested in DOCS-CONSOLE-23-017.
|
||||
- [x] Each tour references authoritative workspace docs and CLI parity commands.
|
||||
- [x] Capture checklist names align with `docs/assets/ui/tours/` convention.
|
||||
- [x] Offline and sealed-mode notes included for every flow.
|
||||
- [x] Security considerations (scopes, fresh-auth, evidence handling) highlighted.
|
||||
- [x] Observability/telemetry pointers surfaced to support Ops follow-up.
|
||||
- [x] Media hygiene guidance documented (assets, compression, versioning).
|
||||
- [x] Document timestamp reflects Sprint 23 delivery.
|
||||
|
||||
---
|
||||
|
||||
*Last updated: 2025-10-27 (Sprint 23).*
|
||||
|
||||
@@ -1,337 +1,337 @@
|
||||
# Export Center REST API
|
||||
|
||||
> **Audience:** Platform integrators, Console/CLI developers, and automation engineers orchestrating export runs.
|
||||
> **Base route:** `/api/export/*` behind the StellaOps gateway; requires Authority-issued tokens with export scopes.
|
||||
|
||||
This reference describes the Export Center API introduced in Export Center Phase 1 (Epic 10) and extended in Phase 2. Use it alongside the [Export Center Architecture](architecture.md) and [Profiles](profiles.md) guides for service-level semantics.
|
||||
|
||||
> Status: Endpoint implementation lands with `EXPORT-SVC-35-006` (Sprint 35) and related follow-on tasks. As of the current build the WebService hosts only the template stub; use this contract for coordination and update once the API is wired.
|
||||
|
||||
## 1. Authentication and headers
|
||||
|
||||
- **Authorization:** Bearer tokens in `Authorization: Bearer <token>` paired with DPoP proof. Required scopes per endpoint:
|
||||
- `export:profile:manage` for profile CRUD.
|
||||
- `export:run` to submit and cancel runs.
|
||||
- `export:read` to list and inspect runs.
|
||||
- `export:download` for bundle downloads and manifests.
|
||||
- **Tenant context:** Provide `X-Stella-Tenant` when the token carries multiple tenants; defaults to token tenant otherwise.
|
||||
- **Idempotency:** Mutating endpoints accept `Idempotency-Key` (UUID). Retrying with the same key returns the original result.
|
||||
- **Rate limits and quotas:** Responses include `X-Stella-Quota-Limit`, `X-Stella-Quota-Remaining`, and `X-Stella-Quota-Reset`. Exceeding quotas returns `429 Too Many Requests` with `ERR_EXPORT_QUOTA`.
|
||||
- **Content negotiation:** Requests and responses use `application/json; charset=utf-8` unless otherwise stated. Downloads stream binary content with profile-specific media types.
|
||||
- **SSE:** Event streams set `Content-Type: text/event-stream` and keep connections alive with comment heartbeats every 15 seconds.
|
||||
|
||||
## 2. Error model
|
||||
|
||||
Errors follow standard HTTP codes with structured payloads:
|
||||
|
||||
```json
|
||||
{
|
||||
"code": "ERR_EXPORT_002",
|
||||
"message": "Profile not found for tenant acme",
|
||||
"details": [],
|
||||
"traceId": "01J9N4Y4K2XY8C5V7T2S",
|
||||
"timestamp": "2025-10-29T13:42:11Z"
|
||||
}
|
||||
```
|
||||
|
||||
| Code | Description | Typical HTTP status | Notes |
|
||||
|------|-------------|---------------------|-------|
|
||||
| `ERR_EXPORT_001` | Validation failure (selectors, configuration) | 400 | `details` enumerates offending fields. |
|
||||
| `ERR_EXPORT_002` | Profile missing or not accessible for tenant | 404 | Returned on run submission or profile fetch. |
|
||||
| `ERR_EXPORT_003` | Concurrency or quota exceeded | 429 | Includes `retryAfterSeconds` in `details`. |
|
||||
| `ERR_EXPORT_004` | Adapter failure (schema mismatch, upstream outage) | 502 | Worker logs contain adapter error reason. |
|
||||
| `ERR_EXPORT_005` | Signing or KMS error | 500 | Run marked failed with `errorCode=signing`. |
|
||||
| `ERR_EXPORT_006` | Distribution failure (HTTP, OCI, object storage) | 502 | `details` lists failing distribution driver. |
|
||||
| `ERR_EXPORT_007` | Run canceled or expired | 409 | Includes cancel author and timestamp. |
|
||||
| `ERR_EXPORT_BASE_MISSING` | Base manifest for delta exports not found | 400 | Specific to `mirror:delta`. |
|
||||
| `ERR_EXPORT_EMPTY` | No records matched selectors (when `allowEmpty=false`) | 422 | Useful for guard-railled automation. |
|
||||
| `ERR_EXPORT_QUOTA` | Daily quota exhausted | 429 | Always paired with quota headers. |
|
||||
|
||||
All responses include `traceId` for correlation with logs and metrics.
|
||||
|
||||
## 3. Profiles endpoints
|
||||
|
||||
### 3.1 List profiles
|
||||
|
||||
```
|
||||
GET /api/export/profiles?kind=json&variant=raw&page=1&pageSize=20
|
||||
Scopes: export:read
|
||||
```
|
||||
|
||||
Returns tenant-scoped profiles. Response headers: `X-Total-Count`, `Link` for pagination.
|
||||
|
||||
**Response**
|
||||
|
||||
```json
|
||||
{
|
||||
"items": [
|
||||
{
|
||||
"profileId": "prof-json-raw",
|
||||
"name": "Daily JSON Raw",
|
||||
"kind": "json",
|
||||
"variant": "raw",
|
||||
"distribution": ["http", "object"],
|
||||
"retention": {"mode": "days", "value": 14},
|
||||
"createdAt": "2025-10-23T08:00:00Z",
|
||||
"createdBy": "user:ops"
|
||||
}
|
||||
],
|
||||
"page": 1,
|
||||
"pageSize": 20
|
||||
}
|
||||
```
|
||||
|
||||
### 3.2 Get a profile
|
||||
|
||||
```
|
||||
GET /api/export/profiles/{profileId}
|
||||
Scopes: export:read
|
||||
```
|
||||
|
||||
Returns full configuration, including `config` payload, distribution options, and metadata.
|
||||
|
||||
### 3.3 Create a profile
|
||||
|
||||
```
|
||||
POST /api/export/profiles
|
||||
Scopes: export:profile:manage
|
||||
```
|
||||
|
||||
**Request**
|
||||
|
||||
```json
|
||||
{
|
||||
"profileId": "prof-airgap-mirror",
|
||||
"name": "Airgap Mirror Weekly",
|
||||
"kind": "mirror",
|
||||
"variant": "full",
|
||||
"include": ["advisories", "vex", "sboms", "policy"],
|
||||
"distribution": ["http", "object"],
|
||||
"encryption": {
|
||||
"enabled": true,
|
||||
"recipientKeys": ["age1tenantkey..."],
|
||||
"strict": false
|
||||
},
|
||||
"retention": {"mode": "days", "value": 30}
|
||||
}
|
||||
```
|
||||
|
||||
**Response 201**
|
||||
|
||||
```json
|
||||
{
|
||||
"profileId": "prof-airgap-mirror",
|
||||
"version": 1,
|
||||
"createdAt": "2025-10-29T12:05:22Z",
|
||||
"createdBy": "user:ops",
|
||||
"status": "active"
|
||||
}
|
||||
```
|
||||
|
||||
### 3.4 Update profile metadata
|
||||
|
||||
```
|
||||
PATCH /api/export/profiles/{profileId}
|
||||
Scopes: export:profile:manage
|
||||
```
|
||||
|
||||
Allows renaming, toggling distribution switches, or updating retention. Structural configuration updates (kind/variant/include) create a new revision; the API returns `revisionCreated=true` and the new `profileId` (e.g., `prof-airgap-mirror@2`).
|
||||
|
||||
### 3.5 Archive profile
|
||||
|
||||
```
|
||||
POST /api/export/profiles/{profileId}:archive
|
||||
Scopes: export:profile:manage
|
||||
```
|
||||
|
||||
Marks profile as inactive; existing runs remain accessible. Use `:restore` to reactivate.
|
||||
|
||||
## 4. Run management
|
||||
|
||||
### 4.1 Submit an export run
|
||||
|
||||
```
|
||||
POST /api/export/runs
|
||||
Scopes: export:run
|
||||
```
|
||||
|
||||
**Request**
|
||||
|
||||
```json
|
||||
{
|
||||
"profileId": "prof-json-raw",
|
||||
"selectors": {
|
||||
"tenants": ["acme"],
|
||||
"timeWindow": {
|
||||
"from": "2025-10-01T00:00:00Z",
|
||||
"to": "2025-10-29T00:00:00Z"
|
||||
},
|
||||
"products": ["registry.example.com/app:*"],
|
||||
"sboms": ["sbom:S-1001", "sbom:S-2004"]
|
||||
},
|
||||
"policySnapshotId": "policy-snap-42",
|
||||
"options": {
|
||||
"allowEmpty": false,
|
||||
"priority": "standard"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Response 202**
|
||||
|
||||
```json
|
||||
{
|
||||
"runId": "run-20251029-01",
|
||||
"status": "pending",
|
||||
"profileId": "prof-json-raw",
|
||||
"createdAt": "2025-10-29T12:12:11Z",
|
||||
"createdBy": "user:ops",
|
||||
"selectors": { "...": "..." },
|
||||
"links": {
|
||||
"self": "/api/export/runs/run-20251029-01",
|
||||
"events": "/api/export/runs/run-20251029-01/events"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 4.2 List runs
|
||||
|
||||
```
|
||||
GET /api/export/runs?status=active&profileId=prof-json-raw&page=1&pageSize=10
|
||||
Scopes: export:read
|
||||
```
|
||||
|
||||
Returns latest runs with pagination. Each item includes summary counts, duration, and last event.
|
||||
|
||||
### 4.3 Get run status
|
||||
|
||||
```
|
||||
GET /api/export/runs/{runId}
|
||||
Scopes: export:read
|
||||
```
|
||||
|
||||
Response fields:
|
||||
|
||||
| Field | Description |
|
||||
|-------|-------------|
|
||||
| `status` | `pending`, `running`, `success`, `failed`, `canceled`. |
|
||||
| `progress` | Object with `adapters`, `bytesWritten`, `recordsProcessed`. |
|
||||
| `errorCode` | Populated when `status=failed` (`signing`, `distribution`, etc). |
|
||||
| `policySnapshotId` | Returned for policy-aware profiles. |
|
||||
| `distributions` | List of available distribution descriptors (type, location, sha256, expiresAt). |
|
||||
|
||||
### 4.4 Cancel a run
|
||||
|
||||
```
|
||||
POST /api/export/runs/{runId}:cancel
|
||||
Scopes: export:run
|
||||
```
|
||||
|
||||
Body optional (`{"reason": "Aborted due to incident INC-123"}`). Returns 202 and pushes `run.canceled` event.
|
||||
|
||||
## 5. Events and telemetry
|
||||
|
||||
### 5.1 Server-sent events
|
||||
|
||||
```
|
||||
GET /api/export/runs/{runId}/events
|
||||
Scopes: export:read
|
||||
Accept: text/event-stream
|
||||
```
|
||||
|
||||
Event payload example:
|
||||
|
||||
```
|
||||
event: run.progress
|
||||
data: {"runId":"run-20251029-01","phase":"adapter","adapter":"json","records":1024,"bytes":7340032,"timestamp":"2025-10-29T12:13:15Z"}
|
||||
```
|
||||
|
||||
Event types:
|
||||
|
||||
| Event | Meaning |
|
||||
|-------|---------|
|
||||
| `run.accepted` | Planner accepted job and queued with Orchestrator. |
|
||||
| `run.progress` | Periodic updates with phase, adapter, counts. |
|
||||
| `run.distribution` | Distribution driver finished (includes descriptor). |
|
||||
| `run.signed` | Signing completed successfully. |
|
||||
| `run.succeeded` | Run marked `success`. |
|
||||
| `run.failed` | Run failed; payload includes `errorCode`. |
|
||||
| `run.canceled` | Run canceled; includes `canceledBy`. |
|
||||
|
||||
SSE heartbeats (`: ping`) keep long-lived connections alive and should be ignored by clients.
|
||||
|
||||
### 5.2 Audit events
|
||||
|
||||
`GET /api/export/runs/{runId}/events?format=audit` returns the same event stream in newline-delimited JSON for offline ingestion.
|
||||
|
||||
## 6. Download endpoints
|
||||
|
||||
### 6.1 Bundle download
|
||||
|
||||
```
|
||||
GET /api/export/runs/{runId}/download
|
||||
Scopes: export:download
|
||||
```
|
||||
|
||||
Streams the primary bundle (tarball, zip, or profile-specific layout). Headers:
|
||||
|
||||
- `Content-Disposition: attachment; filename="export-run-20251029-01.tar.zst"`
|
||||
- `X-Export-Digest: sha256:...`
|
||||
- `X-Export-Size: 73482019`
|
||||
- `X-Export-Encryption: age` (when mirror encryption enabled)
|
||||
|
||||
Supports HTTP range requests for resume functionality. If no bundle exists yet, responds `409` with `ERR_EXPORT_007`.
|
||||
|
||||
### 6.2 Manifest download
|
||||
|
||||
```
|
||||
GET /api/export/runs/{runId}/manifest
|
||||
Scopes: export:download
|
||||
```
|
||||
|
||||
Returns signed `export.json`. To fetch the detached signature, append `?signature=true`.
|
||||
|
||||
### 6.3 Provenance download
|
||||
|
||||
```
|
||||
GET /api/export/runs/{runId}/provenance
|
||||
Scopes: export:download
|
||||
```
|
||||
|
||||
Returns signed `provenance.json`. Supports `?signature=true`. Provenance includes attestation subject digests, policy snapshot ids, adapter versions, and KMS key identifiers.
|
||||
|
||||
### 6.4 Distribution descriptors
|
||||
|
||||
```
|
||||
GET /api/export/runs/{runId}/distributions
|
||||
Scopes: export:read
|
||||
```
|
||||
|
||||
Lists all registered distribution targets (HTTP, OCI, object storage). Each item includes `type`, `location`, `sha256`, `sizeBytes`, and `expiresAt`.
|
||||
|
||||
## 7. Webhook hand-off
|
||||
|
||||
Exports can notify external systems once a run succeeds by registering an HTTP webhook:
|
||||
|
||||
```
|
||||
POST /api/export/webhooks
|
||||
Scopes: export:profile:manage
|
||||
```
|
||||
|
||||
Payload includes `targetUrl`, `events` (e.g., `run.succeeded`), and optional secret for HMAC signatures. Webhook deliveries sign payloads with `X-Stella-Signature` header (`sha256=...`). Retries follow exponential backoff with dead-letter capture in `export_events`.
|
||||
|
||||
## 8. Observability
|
||||
|
||||
- **Metrics endpoint:** `/metrics` (service-local) exposes Prometheus metrics listed in [Architecture](architecture.md#observability).
|
||||
- **Tracing:** When `traceparent` header is provided, worker spans join the calling trace.
|
||||
- **Run lookup by trace:** Use `GET /api/export/runs?traceId={id}` when troubleshooting distributed traces.
|
||||
|
||||
## 9. Related documentation
|
||||
|
||||
- [Export Center Overview](overview.md)
|
||||
- [Export Center Architecture](architecture.md)
|
||||
- [Export Center Profiles](profiles.md)
|
||||
- [Export Center CLI Guide](cli.md) *(companion document)*
|
||||
- [Aggregation-Only Contract reference](../ingestion/aggregation-only-contract.md)
|
||||
|
||||
> **Imposed rule:** Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.
|
||||
# Export Center REST API
|
||||
|
||||
> **Audience:** Platform integrators, Console/CLI developers, and automation engineers orchestrating export runs.
|
||||
> **Base route:** `/api/export/*` behind the StellaOps gateway; requires Authority-issued tokens with export scopes.
|
||||
|
||||
This reference describes the Export Center API introduced in Export Center Phase 1 (Epic 10) and extended in Phase 2. Use it alongside the [Export Center Architecture](architecture.md) and [Profiles](profiles.md) guides for service-level semantics.
|
||||
|
||||
> Status: Endpoint implementation lands with `EXPORT-SVC-35-006` (Sprint 35) and related follow-on tasks. As of the current build the WebService hosts only the template stub; use this contract for coordination and update once the API is wired.
|
||||
|
||||
## 1. Authentication and headers
|
||||
|
||||
- **Authorization:** Bearer tokens in `Authorization: Bearer <token>` paired with DPoP proof. Required scopes per endpoint:
|
||||
- `export:profile:manage` for profile CRUD.
|
||||
- `export:run` to submit and cancel runs.
|
||||
- `export:read` to list and inspect runs.
|
||||
- `export:download` for bundle downloads and manifests.
|
||||
- **Tenant context:** Provide `X-Stella-Tenant` when the token carries multiple tenants; defaults to token tenant otherwise.
|
||||
- **Idempotency:** Mutating endpoints accept `Idempotency-Key` (UUID). Retrying with the same key returns the original result.
|
||||
- **Rate limits and quotas:** Responses include `X-Stella-Quota-Limit`, `X-Stella-Quota-Remaining`, and `X-Stella-Quota-Reset`. Exceeding quotas returns `429 Too Many Requests` with `ERR_EXPORT_QUOTA`.
|
||||
- **Content negotiation:** Requests and responses use `application/json; charset=utf-8` unless otherwise stated. Downloads stream binary content with profile-specific media types.
|
||||
- **SSE:** Event streams set `Content-Type: text/event-stream` and keep connections alive with comment heartbeats every 15 seconds.
|
||||
|
||||
## 2. Error model
|
||||
|
||||
Errors follow standard HTTP codes with structured payloads:
|
||||
|
||||
```json
|
||||
{
|
||||
"code": "ERR_EXPORT_002",
|
||||
"message": "Profile not found for tenant acme",
|
||||
"details": [],
|
||||
"traceId": "01J9N4Y4K2XY8C5V7T2S",
|
||||
"timestamp": "2025-10-29T13:42:11Z"
|
||||
}
|
||||
```
|
||||
|
||||
| Code | Description | Typical HTTP status | Notes |
|
||||
|------|-------------|---------------------|-------|
|
||||
| `ERR_EXPORT_001` | Validation failure (selectors, configuration) | 400 | `details` enumerates offending fields. |
|
||||
| `ERR_EXPORT_002` | Profile missing or not accessible for tenant | 404 | Returned on run submission or profile fetch. |
|
||||
| `ERR_EXPORT_003` | Concurrency or quota exceeded | 429 | Includes `retryAfterSeconds` in `details`. |
|
||||
| `ERR_EXPORT_004` | Adapter failure (schema mismatch, upstream outage) | 502 | Worker logs contain adapter error reason. |
|
||||
| `ERR_EXPORT_005` | Signing or KMS error | 500 | Run marked failed with `errorCode=signing`. |
|
||||
| `ERR_EXPORT_006` | Distribution failure (HTTP, OCI, object storage) | 502 | `details` lists failing distribution driver. |
|
||||
| `ERR_EXPORT_007` | Run canceled or expired | 409 | Includes cancel author and timestamp. |
|
||||
| `ERR_EXPORT_BASE_MISSING` | Base manifest for delta exports not found | 400 | Specific to `mirror:delta`. |
|
||||
| `ERR_EXPORT_EMPTY` | No records matched selectors (when `allowEmpty=false`) | 422 | Useful for guard-railled automation. |
|
||||
| `ERR_EXPORT_QUOTA` | Daily quota exhausted | 429 | Always paired with quota headers. |
|
||||
|
||||
All responses include `traceId` for correlation with logs and metrics.
|
||||
|
||||
## 3. Profiles endpoints
|
||||
|
||||
### 3.1 List profiles
|
||||
|
||||
```
|
||||
GET /api/export/profiles?kind=json&variant=raw&page=1&pageSize=20
|
||||
Scopes: export:read
|
||||
```
|
||||
|
||||
Returns tenant-scoped profiles. Response headers: `X-Total-Count`, `Link` for pagination.
|
||||
|
||||
**Response**
|
||||
|
||||
```json
|
||||
{
|
||||
"items": [
|
||||
{
|
||||
"profileId": "prof-json-raw",
|
||||
"name": "Daily JSON Raw",
|
||||
"kind": "json",
|
||||
"variant": "raw",
|
||||
"distribution": ["http", "object"],
|
||||
"retention": {"mode": "days", "value": 14},
|
||||
"createdAt": "2025-10-23T08:00:00Z",
|
||||
"createdBy": "user:ops"
|
||||
}
|
||||
],
|
||||
"page": 1,
|
||||
"pageSize": 20
|
||||
}
|
||||
```
|
||||
|
||||
### 3.2 Get a profile
|
||||
|
||||
```
|
||||
GET /api/export/profiles/{profileId}
|
||||
Scopes: export:read
|
||||
```
|
||||
|
||||
Returns full configuration, including `config` payload, distribution options, and metadata.
|
||||
|
||||
### 3.3 Create a profile
|
||||
|
||||
```
|
||||
POST /api/export/profiles
|
||||
Scopes: export:profile:manage
|
||||
```
|
||||
|
||||
**Request**
|
||||
|
||||
```json
|
||||
{
|
||||
"profileId": "prof-airgap-mirror",
|
||||
"name": "Airgap Mirror Weekly",
|
||||
"kind": "mirror",
|
||||
"variant": "full",
|
||||
"include": ["advisories", "vex", "sboms", "policy"],
|
||||
"distribution": ["http", "object"],
|
||||
"encryption": {
|
||||
"enabled": true,
|
||||
"recipientKeys": ["age1tenantkey..."],
|
||||
"strict": false
|
||||
},
|
||||
"retention": {"mode": "days", "value": 30}
|
||||
}
|
||||
```
|
||||
|
||||
**Response 201**
|
||||
|
||||
```json
|
||||
{
|
||||
"profileId": "prof-airgap-mirror",
|
||||
"version": 1,
|
||||
"createdAt": "2025-10-29T12:05:22Z",
|
||||
"createdBy": "user:ops",
|
||||
"status": "active"
|
||||
}
|
||||
```
|
||||
|
||||
### 3.4 Update profile metadata
|
||||
|
||||
```
|
||||
PATCH /api/export/profiles/{profileId}
|
||||
Scopes: export:profile:manage
|
||||
```
|
||||
|
||||
Allows renaming, toggling distribution switches, or updating retention. Structural configuration updates (kind/variant/include) create a new revision; the API returns `revisionCreated=true` and the new `profileId` (e.g., `prof-airgap-mirror@2`).
|
||||
|
||||
### 3.5 Archive profile
|
||||
|
||||
```
|
||||
POST /api/export/profiles/{profileId}:archive
|
||||
Scopes: export:profile:manage
|
||||
```
|
||||
|
||||
Marks profile as inactive; existing runs remain accessible. Use `:restore` to reactivate.
|
||||
|
||||
## 4. Run management
|
||||
|
||||
### 4.1 Submit an export run
|
||||
|
||||
```
|
||||
POST /api/export/runs
|
||||
Scopes: export:run
|
||||
```
|
||||
|
||||
**Request**
|
||||
|
||||
```json
|
||||
{
|
||||
"profileId": "prof-json-raw",
|
||||
"selectors": {
|
||||
"tenants": ["acme"],
|
||||
"timeWindow": {
|
||||
"from": "2025-10-01T00:00:00Z",
|
||||
"to": "2025-10-29T00:00:00Z"
|
||||
},
|
||||
"products": ["registry.example.com/app:*"],
|
||||
"sboms": ["sbom:S-1001", "sbom:S-2004"]
|
||||
},
|
||||
"policySnapshotId": "policy-snap-42",
|
||||
"options": {
|
||||
"allowEmpty": false,
|
||||
"priority": "standard"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Response 202**
|
||||
|
||||
```json
|
||||
{
|
||||
"runId": "run-20251029-01",
|
||||
"status": "pending",
|
||||
"profileId": "prof-json-raw",
|
||||
"createdAt": "2025-10-29T12:12:11Z",
|
||||
"createdBy": "user:ops",
|
||||
"selectors": { "...": "..." },
|
||||
"links": {
|
||||
"self": "/api/export/runs/run-20251029-01",
|
||||
"events": "/api/export/runs/run-20251029-01/events"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 4.2 List runs
|
||||
|
||||
```
|
||||
GET /api/export/runs?status=active&profileId=prof-json-raw&page=1&pageSize=10
|
||||
Scopes: export:read
|
||||
```
|
||||
|
||||
Returns latest runs with pagination. Each item includes summary counts, duration, and last event.
|
||||
|
||||
### 4.3 Get run status
|
||||
|
||||
```
|
||||
GET /api/export/runs/{runId}
|
||||
Scopes: export:read
|
||||
```
|
||||
|
||||
Response fields:
|
||||
|
||||
| Field | Description |
|
||||
|-------|-------------|
|
||||
| `status` | `pending`, `running`, `success`, `failed`, `canceled`. |
|
||||
| `progress` | Object with `adapters`, `bytesWritten`, `recordsProcessed`. |
|
||||
| `errorCode` | Populated when `status=failed` (`signing`, `distribution`, etc). |
|
||||
| `policySnapshotId` | Returned for policy-aware profiles. |
|
||||
| `distributions` | List of available distribution descriptors (type, location, sha256, expiresAt). |
|
||||
|
||||
### 4.4 Cancel a run
|
||||
|
||||
```
|
||||
POST /api/export/runs/{runId}:cancel
|
||||
Scopes: export:run
|
||||
```
|
||||
|
||||
Body optional (`{"reason": "Aborted due to incident INC-123"}`). Returns 202 and pushes `run.canceled` event.
|
||||
|
||||
## 5. Events and telemetry
|
||||
|
||||
### 5.1 Server-sent events
|
||||
|
||||
```
|
||||
GET /api/export/runs/{runId}/events
|
||||
Scopes: export:read
|
||||
Accept: text/event-stream
|
||||
```
|
||||
|
||||
Event payload example:
|
||||
|
||||
```
|
||||
event: run.progress
|
||||
data: {"runId":"run-20251029-01","phase":"adapter","adapter":"json","records":1024,"bytes":7340032,"timestamp":"2025-10-29T12:13:15Z"}
|
||||
```
|
||||
|
||||
Event types:
|
||||
|
||||
| Event | Meaning |
|
||||
|-------|---------|
|
||||
| `run.accepted` | Planner accepted job and queued with Orchestrator. |
|
||||
| `run.progress` | Periodic updates with phase, adapter, counts. |
|
||||
| `run.distribution` | Distribution driver finished (includes descriptor). |
|
||||
| `run.signed` | Signing completed successfully. |
|
||||
| `run.succeeded` | Run marked `success`. |
|
||||
| `run.failed` | Run failed; payload includes `errorCode`. |
|
||||
| `run.canceled` | Run canceled; includes `canceledBy`. |
|
||||
|
||||
SSE heartbeats (`: ping`) keep long-lived connections alive and should be ignored by clients.
|
||||
|
||||
### 5.2 Audit events
|
||||
|
||||
`GET /api/export/runs/{runId}/events?format=audit` returns the same event stream in newline-delimited JSON for offline ingestion.
|
||||
|
||||
## 6. Download endpoints
|
||||
|
||||
### 6.1 Bundle download
|
||||
|
||||
```
|
||||
GET /api/export/runs/{runId}/download
|
||||
Scopes: export:download
|
||||
```
|
||||
|
||||
Streams the primary bundle (tarball, zip, or profile-specific layout). Headers:
|
||||
|
||||
- `Content-Disposition: attachment; filename="export-run-20251029-01.tar.zst"`
|
||||
- `X-Export-Digest: sha256:...`
|
||||
- `X-Export-Size: 73482019`
|
||||
- `X-Export-Encryption: age` (when mirror encryption enabled)
|
||||
|
||||
Supports HTTP range requests for resume functionality. If no bundle exists yet, responds `409` with `ERR_EXPORT_007`.
|
||||
|
||||
### 6.2 Manifest download
|
||||
|
||||
```
|
||||
GET /api/export/runs/{runId}/manifest
|
||||
Scopes: export:download
|
||||
```
|
||||
|
||||
Returns signed `export.json`. To fetch the detached signature, append `?signature=true`.
|
||||
|
||||
### 6.3 Provenance download
|
||||
|
||||
```
|
||||
GET /api/export/runs/{runId}/provenance
|
||||
Scopes: export:download
|
||||
```
|
||||
|
||||
Returns signed `provenance.json`. Supports `?signature=true`. Provenance includes attestation subject digests, policy snapshot ids, adapter versions, and KMS key identifiers.
|
||||
|
||||
### 6.4 Distribution descriptors
|
||||
|
||||
```
|
||||
GET /api/export/runs/{runId}/distributions
|
||||
Scopes: export:read
|
||||
```
|
||||
|
||||
Lists all registered distribution targets (HTTP, OCI, object storage). Each item includes `type`, `location`, `sha256`, `sizeBytes`, and `expiresAt`.
|
||||
|
||||
## 7. Webhook hand-off
|
||||
|
||||
Exports can notify external systems once a run succeeds by registering an HTTP webhook:
|
||||
|
||||
```
|
||||
POST /api/export/webhooks
|
||||
Scopes: export:profile:manage
|
||||
```
|
||||
|
||||
Payload includes `targetUrl`, `events` (e.g., `run.succeeded`), and optional secret for HMAC signatures. Webhook deliveries sign payloads with `X-Stella-Signature` header (`sha256=...`). Retries follow exponential backoff with dead-letter capture in `export_events`.
|
||||
|
||||
## 8. Observability
|
||||
|
||||
- **Metrics endpoint:** `/metrics` (service-local) exposes Prometheus metrics listed in [Architecture](architecture.md#observability).
|
||||
- **Tracing:** When `traceparent` header is provided, worker spans join the calling trace.
|
||||
- **Run lookup by trace:** Use `GET /api/export/runs?traceId={id}` when troubleshooting distributed traces.
|
||||
|
||||
## 9. Related documentation
|
||||
|
||||
- [Export Center Overview](overview.md)
|
||||
- [Export Center Architecture](architecture.md)
|
||||
- [Export Center Profiles](profiles.md)
|
||||
- [Export Center CLI Guide](cli.md) *(companion document)*
|
||||
- [Aggregation-Only Contract reference](../ingestion/aggregation-only-contract.md)
|
||||
|
||||
> **Imposed rule:** Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.
|
||||
|
||||
@@ -1,125 +1,125 @@
|
||||
# Export Center Architecture
|
||||
|
||||
The Export Center is the dedicated service layer that packages StellaOps evidence and policy overlays into reproducible bundles. It runs as a multi-surface API backed by asynchronous workers and format adapters, enforcing Aggregation-Only Contract (AOC) guardrails while providing deterministic manifests, signing, and distribution paths.
|
||||
|
||||
## Runtime topology
|
||||
- **Export Center API (`StellaOps.ExportCenter.WebService`).** Receives profile CRUD, export run requests, status queries, and download streams through the unified Web API gateway. Enforces tenant scopes, RBAC, quotas, and concurrency guards.
|
||||
- **Export Center Worker (`StellaOps.ExportCenter.Worker`).** Dequeues export jobs from the Orchestrator, resolves selectors, invokes adapters, and writes manifests and bundle artefacts. Stateless; scales horizontally.
|
||||
- **Backing stores.**
|
||||
- MongoDB collections: `export_profiles`, `export_runs`, `export_inputs`, `export_distributions`, `export_events`.
|
||||
- Object storage bucket or filesystem for staging bundle payloads.
|
||||
- Optional registry/object storage credentials injected via Authority-scoped secrets.
|
||||
- **Integration peers.**
|
||||
- **Findings Ledger** for advisory, VEX, SBOM payload streaming.
|
||||
- **Policy Engine** for deterministic policy snapshots and evaluated findings.
|
||||
- **Orchestrator** for job scheduling, quotas, and telemetry fan-out.
|
||||
- **Authority** for tenant-aware access tokens and KMS key references.
|
||||
- **Console & CLI** as presentation surfaces consuming the API.
|
||||
|
||||
## Job lifecycle
|
||||
1. **Profile selection.** Operator or automation picks a profile (`json:raw`, `json:policy`, `trivy:db`, `trivy:java-db`, `mirror:full`, `mirror:delta`) and submits scope selectors (tenant, time window, products, SBOM subjects, ecosystems). See `docs/export-center/profiles.md` for profile definitions and configuration fields.
|
||||
2. **Planner resolution.** API validates selectors, expands include/exclude lists, and writes a pending `export_run` with immutable parameters and deterministic ordering hints.
|
||||
3. **Orchestrator dispatch.** `export_run` triggers a job lease via Orchestrator with quotas per tenant/profile and concurrency caps (default 4 active per tenant).
|
||||
4. **Worker execution.** Worker streams data from Findings Ledger and Policy Engine using pagination cursors. Adapters write canonical payloads to staging storage, compute checksums, and emit streaming progress events (SSE).
|
||||
5. **Manifest and provenance emission.** Worker writes `export.json` and `provenance.json`, signs them with configured KMS keys (cosign-compatible), and uploads signatures alongside content.
|
||||
6. **Distribution registration.** Worker records available distribution methods (download URL, OCI reference, object storage path), raises completion/failure events, and exposes metrics/logs.
|
||||
7. **Download & verification.** Clients download bundles or pull OCI artefacts, verify signatures, and consume provenance to trace source artefacts.
|
||||
|
||||
Cancellation requests mark runs as `aborted` and cause workers to stop iterating sources; partially written files are destroyed and the run is marked with an audit entry.
|
||||
|
||||
## Core components
|
||||
### API surface
|
||||
- Detailed request and response payloads are catalogued in `docs/export-center/api.md`.
|
||||
- **Profiles API.**
|
||||
- `GET /api/export/profiles`: list tenant-scoped profiles.
|
||||
- `POST /api/export/profiles`: create custom profiles (variants of JSON, Trivy, mirror) with validated configuration schema.
|
||||
- `PATCH /api/export/profiles/{id}`: update metadata; config changes clone new revision to preserve determinism.
|
||||
- **Runs API.**
|
||||
- `POST /api/export/runs`: submit export run for a profile with selectors and options (policy snapshot id, mirror base manifest).
|
||||
- `GET /api/export/runs/{id}`: status, progress counters, provenance summary.
|
||||
- `GET /api/export/runs/{id}/events`: server-sent events with state transitions, adapter milestones, signing status.
|
||||
- `POST /api/export/runs/{id}/cancel`: cooperative cancellation with audit logging.
|
||||
- **Downloads API.**
|
||||
- `GET /api/export/runs/{id}/download`: streaming download with range support and checksum trailers.
|
||||
- `GET /api/export/runs/{id}/manifest`: signed `export.json`.
|
||||
- `GET /api/export/runs/{id}/provenance`: signed `provenance.json`.
|
||||
|
||||
All endpoints require Authority-issued JWT + DPoP tokens with scopes `export:run`, `export:read`, and tenant claim alignment. Rate-limiting and quotas surface via `X-Stella-Quota-*` headers.
|
||||
|
||||
### Worker pipeline
|
||||
- **Input resolvers.** Query Findings Ledger and Policy Engine using stable pagination (Mongo `_id` ascending, or resume tokens for change streams). Selector expressions compile into Mongo filter fragments and/or API query parameters.
|
||||
- **Adapter host.** Adapter plugin loader (restart-time only) resolves profile variant to adapter implementation. Adapters present a deterministic `RunAsync(context)` contract with streaming writers and telemetry instrumentation.
|
||||
- **Content writers.**
|
||||
- JSON adapters emit `.jsonl.zst` files with canonical ordering (tenant, subject, document id).
|
||||
- Trivy adapters materialise SQLite databases or tar archives matching Trivy DB expectations; schema version gates prevent unsupported outputs.
|
||||
- Mirror adapters assemble deterministic filesystem trees (manifests, indexes, payload subtrees) and, when configured, OCI artefact layers.
|
||||
- **Manifest generator.** Aggregates counts, bytes, hash digests (SHA-256), profile metadata, and input references. Writes `export.json` and `provenance.json` using canonical JSON (sorted keys, RFC3339 UTC timestamps).
|
||||
- **Signing service.** Integrates with platform KMS via Authority (default cosign signer). Produces in-toto SLSA attestations when configured. Supports detached signatures and optional in-bundle signatures.
|
||||
- **Distribution drivers.** `dist-http` exposes staged files via download endpoint; `dist-oci` pushes artefacts to registries using ORAS with digest pinning; `dist-objstore` uploads to tenant-specific prefixes with immutability flags.
|
||||
|
||||
## Data model snapshots
|
||||
|
||||
| Collection | Purpose | Key fields | Notes |
|
||||
|------------|---------|------------|-------|
|
||||
| `export_profiles` | Profile definitions (kind, variant, config). | `_id`, `tenant`, `name`, `kind`, `variant`, `config_json`, `created_by`, `created_at`. | Config includes adapter parameters (included record types, compression, encryption). |
|
||||
| `export_runs` | Run state machine and audit info. | `_id`, `profile_id`, `tenant`, `status`, `requested_by`, `selectors`, `policy_snapshot_id`, `started_at`, `completed_at`, `duration_ms`, `error_code`. | Immutable selectors; status transitions recorded in `export_events`. |
|
||||
| `export_inputs` | Resolved input ranges. | `run_id`, `source`, `cursor`, `count`, `hash`. | Enables resumable retries and audit. |
|
||||
| `export_distributions` | Distribution artefacts. | `run_id`, `type` (`http`, `oci`, `object`), `location`, `sha256`, `size_bytes`, `expires_at`. | `expires_at` used for retention policies and automatic pruning. |
|
||||
| `export_events` | Timeline of state transitions and metrics. | `run_id`, `event_type`, `message`, `at`, `metrics`. | Feeds SSE stream and audit trails. |
|
||||
|
||||
## Adapter responsibilities
|
||||
- **JSON (`json:raw`, `json:policy`).**
|
||||
- Ensures canonical casing, timezone normalization, and linkset preservation.
|
||||
- Policy variant embeds policy snapshot metadata (`policy_version`, `inputs_hash`, `decision_trace` fingerprint) and emits evaluated findings as separate files.
|
||||
- Enforces AOC guardrails: no derived modifications to raw evidence fields.
|
||||
- **Trivy (`trivy:db`, `trivy:java-db`).**
|
||||
- Maps StellaOps advisory schema to Trivy DB format, handling namespace collisions and ecosystem-specific ranges.
|
||||
- Validates compatibility against supported Trivy schema versions; run fails fast if mismatch.
|
||||
- Emits optional manifest summarising package counts and severity distribution.
|
||||
- **Mirror (`mirror:full`, `mirror:delta`).**
|
||||
- Builds self-contained filesystem layout (`/manifests`, `/data/raw`, `/data/policy`, `/indexes`).
|
||||
- Delta variant compares against base manifest (`base_export_id`) to write only changed artefacts; records `removed` entries for cleanup.
|
||||
- Supports optional encryption of `/data` subtree (age/AES-GCM) with key wrapping stored in `provenance.json`.
|
||||
|
||||
Adapters expose structured telemetry events (`adapter.start`, `adapter.chunk`, `adapter.complete`) with record counts and byte totals per chunk. Failures emit `adapter.error` with reason codes.
|
||||
|
||||
## Signing and provenance
|
||||
- **Manifest schema.** `export.json` contains run metadata, profile descriptor, selector summary, counts, SHA-256 digests, compression hints, and distribution list. Deterministic field ordering and normalized timestamps.
|
||||
- **Provenance schema.** `provenance.json` captures in-toto subject listing (bundle digest, manifest digest), referenced inputs (findings ledger queries, policy snapshot ids, SBOM identifiers), tool version (`exporter_version`, adapter versions), and KMS key identifiers.
|
||||
- **Attestation.** Cosign SLSA Level 2 template by default; optional SLSA Level 3 when supply chain attestations are enabled. Detached signatures stored alongside manifests; CLI/Console encourage `cosign verify --key <tenant-key>` workflow.
|
||||
- **Audit trail.** Each run stores success/failure status, signature identifiers, and verification hints for downstream automation (CI pipelines, offline verification scripts).
|
||||
|
||||
## Distribution flows
|
||||
- **HTTP download.** Console and CLI stream bundles via chunked transfer; supports range requests and resumable downloads. Response includes `X-Export-Digest`, `X-Export-Length`, and optional encryption metadata.
|
||||
- **OCI push.** Worker uses ORAS to publish bundles as OCI artefacts with annotations describing profile, tenant, manifest digest, and provenance reference. Supports multi-tenant registries with `repository-per-tenant` naming.
|
||||
- **Object storage.** Writes to tenant-prefixed paths (`s3://stella-exports/{tenant}/{run-id}/...`) with immutable retention policies. Retention scheduler purges expired runs based on profile configuration.
|
||||
- **Offline Kit seeding.** Mirror bundles optionally staged into Offline Kit assembly pipelines, inheriting the same manifests and signatures.
|
||||
|
||||
## Observability
|
||||
- **Metrics.** Emits `exporter_run_duration_seconds`, `exporter_run_bytes_total{profile}`, `exporter_run_failures_total{error_code}`, `exporter_active_runs{tenant}`, `exporter_distribution_push_seconds{type}`.
|
||||
- **Logs.** Structured logs with fields `run_id`, `tenant`, `profile_kind`, `adapter`, `phase`, `correlation_id`, `error_code`. Phases include `plan`, `resolve`, `adapter`, `manifest`, `sign`, `distribute`.
|
||||
- **Traces.** Optional OpenTelemetry spans (`export.plan`, `export.fetch`, `export.write`, `export.sign`, `export.distribute`) for cross-service correlation.
|
||||
- **Dashboards & alerts.** DevOps pipeline seeds Grafana dashboards summarising throughput, size, failure ratios, and distribution latency. Alert thresholds: failure rate >5% per profile, median run duration >p95 baseline, signature verification failures >0.
|
||||
|
||||
## Security posture
|
||||
- Tenant claim enforced at every query and distribution path; cross-tenant selectors rejected unless explicit cross-tenant mirror feature toggled with signed approval.
|
||||
- RBAC scopes: `export:profile:manage`, `export:run`, `export:read`, `export:download`. Console hides actions without scope; CLI returns `401/403`.
|
||||
- Encryption options configurable per profile; keys derived from Authority-managed KMS. Mirror encryption uses tenant-specific recipients; JSON/Trivy rely on transport security plus optional encryption at rest.
|
||||
- Restart-only plugin loading ensures adapters and distribution drivers are vetted at deployment time, reducing runtime injection risks.
|
||||
- Deterministic output ensures tamper detection via content hashes; provenance links to source runs and policy snapshots to maintain auditability.
|
||||
|
||||
## Deployment considerations
|
||||
- Packaged as separate API and worker containers. Helm chart and compose overlays define horizontal scaling, worker concurrency, queue leases, and object storage credentials.
|
||||
- Requires Authority client credentials for KMS and optional registry credentials stored via sealed secrets.
|
||||
- Offline-first deployments disable OCI distribution by default and provide local object storage endpoints; HTTP downloads served via internal gateway.
|
||||
- Health endpoints: `/health/ready` validates Mongo connectivity, object storage access, adapter registry integrity, and KMS signer readiness.
|
||||
|
||||
## Compliance checklist
|
||||
- [ ] Profiles and runs enforce tenant scoping; cross-tenant exports disabled unless approved.
|
||||
- [ ] Manifests and provenance files are generated with deterministic hashes and signed via configured KMS.
|
||||
- [ ] Adapters run with restart-time registration only; no runtime plugin loading.
|
||||
- [ ] Distribution drivers respect allowlist; OCI push disabled when offline mode is active.
|
||||
- [ ] Metrics, logs, and traces follow observability guidelines; dashboards and alerts configured.
|
||||
- [ ] Retention policies and pruning jobs configured for staged bundles.
|
||||
|
||||
> **Imposed rule:** Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.
|
||||
# Export Center Architecture
|
||||
|
||||
The Export Center is the dedicated service layer that packages StellaOps evidence and policy overlays into reproducible bundles. It runs as a multi-surface API backed by asynchronous workers and format adapters, enforcing Aggregation-Only Contract (AOC) guardrails while providing deterministic manifests, signing, and distribution paths.
|
||||
|
||||
## Runtime topology
|
||||
- **Export Center API (`StellaOps.ExportCenter.WebService`).** Receives profile CRUD, export run requests, status queries, and download streams through the unified Web API gateway. Enforces tenant scopes, RBAC, quotas, and concurrency guards.
|
||||
- **Export Center Worker (`StellaOps.ExportCenter.Worker`).** Dequeues export jobs from the Orchestrator, resolves selectors, invokes adapters, and writes manifests and bundle artefacts. Stateless; scales horizontally.
|
||||
- **Backing stores.**
|
||||
- MongoDB collections: `export_profiles`, `export_runs`, `export_inputs`, `export_distributions`, `export_events`.
|
||||
- Object storage bucket or filesystem for staging bundle payloads.
|
||||
- Optional registry/object storage credentials injected via Authority-scoped secrets.
|
||||
- **Integration peers.**
|
||||
- **Findings Ledger** for advisory, VEX, SBOM payload streaming.
|
||||
- **Policy Engine** for deterministic policy snapshots and evaluated findings.
|
||||
- **Orchestrator** for job scheduling, quotas, and telemetry fan-out.
|
||||
- **Authority** for tenant-aware access tokens and KMS key references.
|
||||
- **Console & CLI** as presentation surfaces consuming the API.
|
||||
|
||||
## Job lifecycle
|
||||
1. **Profile selection.** Operator or automation picks a profile (`json:raw`, `json:policy`, `trivy:db`, `trivy:java-db`, `mirror:full`, `mirror:delta`) and submits scope selectors (tenant, time window, products, SBOM subjects, ecosystems). See `docs/export-center/profiles.md` for profile definitions and configuration fields.
|
||||
2. **Planner resolution.** API validates selectors, expands include/exclude lists, and writes a pending `export_run` with immutable parameters and deterministic ordering hints.
|
||||
3. **Orchestrator dispatch.** `export_run` triggers a job lease via Orchestrator with quotas per tenant/profile and concurrency caps (default 4 active per tenant).
|
||||
4. **Worker execution.** Worker streams data from Findings Ledger and Policy Engine using pagination cursors. Adapters write canonical payloads to staging storage, compute checksums, and emit streaming progress events (SSE).
|
||||
5. **Manifest and provenance emission.** Worker writes `export.json` and `provenance.json`, signs them with configured KMS keys (cosign-compatible), and uploads signatures alongside content.
|
||||
6. **Distribution registration.** Worker records available distribution methods (download URL, OCI reference, object storage path), raises completion/failure events, and exposes metrics/logs.
|
||||
7. **Download & verification.** Clients download bundles or pull OCI artefacts, verify signatures, and consume provenance to trace source artefacts.
|
||||
|
||||
Cancellation requests mark runs as `aborted` and cause workers to stop iterating sources; partially written files are destroyed and the run is marked with an audit entry.
|
||||
|
||||
## Core components
|
||||
### API surface
|
||||
- Detailed request and response payloads are catalogued in `docs/export-center/api.md`.
|
||||
- **Profiles API.**
|
||||
- `GET /api/export/profiles`: list tenant-scoped profiles.
|
||||
- `POST /api/export/profiles`: create custom profiles (variants of JSON, Trivy, mirror) with validated configuration schema.
|
||||
- `PATCH /api/export/profiles/{id}`: update metadata; config changes clone new revision to preserve determinism.
|
||||
- **Runs API.**
|
||||
- `POST /api/export/runs`: submit export run for a profile with selectors and options (policy snapshot id, mirror base manifest).
|
||||
- `GET /api/export/runs/{id}`: status, progress counters, provenance summary.
|
||||
- `GET /api/export/runs/{id}/events`: server-sent events with state transitions, adapter milestones, signing status.
|
||||
- `POST /api/export/runs/{id}/cancel`: cooperative cancellation with audit logging.
|
||||
- **Downloads API.**
|
||||
- `GET /api/export/runs/{id}/download`: streaming download with range support and checksum trailers.
|
||||
- `GET /api/export/runs/{id}/manifest`: signed `export.json`.
|
||||
- `GET /api/export/runs/{id}/provenance`: signed `provenance.json`.
|
||||
|
||||
All endpoints require Authority-issued JWT + DPoP tokens with scopes `export:run`, `export:read`, and tenant claim alignment. Rate-limiting and quotas surface via `X-Stella-Quota-*` headers.
|
||||
|
||||
### Worker pipeline
|
||||
- **Input resolvers.** Query Findings Ledger and Policy Engine using stable pagination (Mongo `_id` ascending, or resume tokens for change streams). Selector expressions compile into Mongo filter fragments and/or API query parameters.
|
||||
- **Adapter host.** Adapter plugin loader (restart-time only) resolves profile variant to adapter implementation. Adapters present a deterministic `RunAsync(context)` contract with streaming writers and telemetry instrumentation.
|
||||
- **Content writers.**
|
||||
- JSON adapters emit `.jsonl.zst` files with canonical ordering (tenant, subject, document id).
|
||||
- Trivy adapters materialise SQLite databases or tar archives matching Trivy DB expectations; schema version gates prevent unsupported outputs.
|
||||
- Mirror adapters assemble deterministic filesystem trees (manifests, indexes, payload subtrees) and, when configured, OCI artefact layers.
|
||||
- **Manifest generator.** Aggregates counts, bytes, hash digests (SHA-256), profile metadata, and input references. Writes `export.json` and `provenance.json` using canonical JSON (sorted keys, RFC3339 UTC timestamps).
|
||||
- **Signing service.** Integrates with platform KMS via Authority (default cosign signer). Produces in-toto SLSA attestations when configured. Supports detached signatures and optional in-bundle signatures.
|
||||
- **Distribution drivers.** `dist-http` exposes staged files via download endpoint; `dist-oci` pushes artefacts to registries using ORAS with digest pinning; `dist-objstore` uploads to tenant-specific prefixes with immutability flags.
|
||||
|
||||
## Data model snapshots
|
||||
|
||||
| Collection | Purpose | Key fields | Notes |
|
||||
|------------|---------|------------|-------|
|
||||
| `export_profiles` | Profile definitions (kind, variant, config). | `_id`, `tenant`, `name`, `kind`, `variant`, `config_json`, `created_by`, `created_at`. | Config includes adapter parameters (included record types, compression, encryption). |
|
||||
| `export_runs` | Run state machine and audit info. | `_id`, `profile_id`, `tenant`, `status`, `requested_by`, `selectors`, `policy_snapshot_id`, `started_at`, `completed_at`, `duration_ms`, `error_code`. | Immutable selectors; status transitions recorded in `export_events`. |
|
||||
| `export_inputs` | Resolved input ranges. | `run_id`, `source`, `cursor`, `count`, `hash`. | Enables resumable retries and audit. |
|
||||
| `export_distributions` | Distribution artefacts. | `run_id`, `type` (`http`, `oci`, `object`), `location`, `sha256`, `size_bytes`, `expires_at`. | `expires_at` used for retention policies and automatic pruning. |
|
||||
| `export_events` | Timeline of state transitions and metrics. | `run_id`, `event_type`, `message`, `at`, `metrics`. | Feeds SSE stream and audit trails. |
|
||||
|
||||
## Adapter responsibilities
|
||||
- **JSON (`json:raw`, `json:policy`).**
|
||||
- Ensures canonical casing, timezone normalization, and linkset preservation.
|
||||
- Policy variant embeds policy snapshot metadata (`policy_version`, `inputs_hash`, `decision_trace` fingerprint) and emits evaluated findings as separate files.
|
||||
- Enforces AOC guardrails: no derived modifications to raw evidence fields.
|
||||
- **Trivy (`trivy:db`, `trivy:java-db`).**
|
||||
- Maps StellaOps advisory schema to Trivy DB format, handling namespace collisions and ecosystem-specific ranges.
|
||||
- Validates compatibility against supported Trivy schema versions; run fails fast if mismatch.
|
||||
- Emits optional manifest summarising package counts and severity distribution.
|
||||
- **Mirror (`mirror:full`, `mirror:delta`).**
|
||||
- Builds self-contained filesystem layout (`/manifests`, `/data/raw`, `/data/policy`, `/indexes`).
|
||||
- Delta variant compares against base manifest (`base_export_id`) to write only changed artefacts; records `removed` entries for cleanup.
|
||||
- Supports optional encryption of `/data` subtree (age/AES-GCM) with key wrapping stored in `provenance.json`.
|
||||
|
||||
Adapters expose structured telemetry events (`adapter.start`, `adapter.chunk`, `adapter.complete`) with record counts and byte totals per chunk. Failures emit `adapter.error` with reason codes.
|
||||
|
||||
## Signing and provenance
|
||||
- **Manifest schema.** `export.json` contains run metadata, profile descriptor, selector summary, counts, SHA-256 digests, compression hints, and distribution list. Deterministic field ordering and normalized timestamps.
|
||||
- **Provenance schema.** `provenance.json` captures in-toto subject listing (bundle digest, manifest digest), referenced inputs (findings ledger queries, policy snapshot ids, SBOM identifiers), tool version (`exporter_version`, adapter versions), and KMS key identifiers.
|
||||
- **Attestation.** Cosign SLSA Level 2 template by default; optional SLSA Level 3 when supply chain attestations are enabled. Detached signatures stored alongside manifests; CLI/Console encourage `cosign verify --key <tenant-key>` workflow.
|
||||
- **Audit trail.** Each run stores success/failure status, signature identifiers, and verification hints for downstream automation (CI pipelines, offline verification scripts).
|
||||
|
||||
## Distribution flows
|
||||
- **HTTP download.** Console and CLI stream bundles via chunked transfer; supports range requests and resumable downloads. Response includes `X-Export-Digest`, `X-Export-Length`, and optional encryption metadata.
|
||||
- **OCI push.** Worker uses ORAS to publish bundles as OCI artefacts with annotations describing profile, tenant, manifest digest, and provenance reference. Supports multi-tenant registries with `repository-per-tenant` naming.
|
||||
- **Object storage.** Writes to tenant-prefixed paths (`s3://stella-exports/{tenant}/{run-id}/...`) with immutable retention policies. Retention scheduler purges expired runs based on profile configuration.
|
||||
- **Offline Kit seeding.** Mirror bundles optionally staged into Offline Kit assembly pipelines, inheriting the same manifests and signatures.
|
||||
|
||||
## Observability
|
||||
- **Metrics.** Emits `exporter_run_duration_seconds`, `exporter_run_bytes_total{profile}`, `exporter_run_failures_total{error_code}`, `exporter_active_runs{tenant}`, `exporter_distribution_push_seconds{type}`.
|
||||
- **Logs.** Structured logs with fields `run_id`, `tenant`, `profile_kind`, `adapter`, `phase`, `correlation_id`, `error_code`. Phases include `plan`, `resolve`, `adapter`, `manifest`, `sign`, `distribute`.
|
||||
- **Traces.** Optional OpenTelemetry spans (`export.plan`, `export.fetch`, `export.write`, `export.sign`, `export.distribute`) for cross-service correlation.
|
||||
- **Dashboards & alerts.** DevOps pipeline seeds Grafana dashboards summarising throughput, size, failure ratios, and distribution latency. Alert thresholds: failure rate >5% per profile, median run duration >p95 baseline, signature verification failures >0.
|
||||
|
||||
## Security posture
|
||||
- Tenant claim enforced at every query and distribution path; cross-tenant selectors rejected unless explicit cross-tenant mirror feature toggled with signed approval.
|
||||
- RBAC scopes: `export:profile:manage`, `export:run`, `export:read`, `export:download`. Console hides actions without scope; CLI returns `401/403`.
|
||||
- Encryption options configurable per profile; keys derived from Authority-managed KMS. Mirror encryption uses tenant-specific recipients; JSON/Trivy rely on transport security plus optional encryption at rest.
|
||||
- Restart-only plugin loading ensures adapters and distribution drivers are vetted at deployment time, reducing runtime injection risks.
|
||||
- Deterministic output ensures tamper detection via content hashes; provenance links to source runs and policy snapshots to maintain auditability.
|
||||
|
||||
## Deployment considerations
|
||||
- Packaged as separate API and worker containers. Helm chart and compose overlays define horizontal scaling, worker concurrency, queue leases, and object storage credentials.
|
||||
- Requires Authority client credentials for KMS and optional registry credentials stored via sealed secrets.
|
||||
- Offline-first deployments disable OCI distribution by default and provide local object storage endpoints; HTTP downloads served via internal gateway.
|
||||
- Health endpoints: `/health/ready` validates Mongo connectivity, object storage access, adapter registry integrity, and KMS signer readiness.
|
||||
|
||||
## Compliance checklist
|
||||
- [ ] Profiles and runs enforce tenant scoping; cross-tenant exports disabled unless approved.
|
||||
- [ ] Manifests and provenance files are generated with deterministic hashes and signed via configured KMS.
|
||||
- [ ] Adapters run with restart-time registration only; no runtime plugin loading.
|
||||
- [ ] Distribution drivers respect allowlist; OCI push disabled when offline mode is active.
|
||||
- [ ] Metrics, logs, and traces follow observability guidelines; dashboards and alerts configured.
|
||||
- [ ] Retention policies and pruning jobs configured for staged bundles.
|
||||
|
||||
> **Imposed rule:** Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.
|
||||
|
||||
@@ -1,231 +1,231 @@
|
||||
# Stella CLI - Export Center Commands
|
||||
|
||||
> **Audience:** Operators, release engineers, and CI maintainers using the `stella` CLI to manage Export Center profiles and runs.
|
||||
> **Supported from:** `stella` CLI >= 0.22.0 (Export Center Phase 1).
|
||||
> **Prerequisites:** Authority token with the scopes noted per command (`export:profile:manage`, `export:run`, `export:read`, `export:download`).
|
||||
|
||||
Use this guide with the [Export Center API reference](api.md) and [Profiles](profiles.md) catalogue. The CLI wraps the same REST endpoints, preserving deterministic behaviour and guardrails.
|
||||
|
||||
> Status: CLI support is tracked under `CLI-EXPORT-35-001` and `CLI-EXPORT-36-001`. The current CLI build does not yet surface these commands; treat this guide as the target contract and adjust once implementations merge.
|
||||
|
||||
## 1. Global options and configuration
|
||||
|
||||
| Flag | Default | Description |
|
||||
|------|---------|-------------|
|
||||
| `--server <url>` | `https://stella.local` | Gateway root. Matches `STELLA_SERVER`. |
|
||||
| `--tenant <id>` | Token tenant | Override tenant for multi-tenant tokens. |
|
||||
| `--profile <name>` | none | Loads saved defaults from `~/.stella/profiles/<name>.toml`. |
|
||||
| `--output <file>` | stdout | Redirect full JSON response. |
|
||||
| `--format <table|json|yaml>` | `table` on TTY | Controls table formatting for list commands. |
|
||||
| `--trace` | false | Emit request timing and correlation ids. |
|
||||
|
||||
Environment variables: `STELLA_TOKEN`, `STELLA_SERVER`, `STELLA_TENANT`, `STELLA_PROFILE`.
|
||||
|
||||
Exit codes align with API error codes (see section 6).
|
||||
|
||||
## 2. Profile management commands
|
||||
|
||||
### 2.1 `stella export profile list`
|
||||
|
||||
List profiles for the current tenant.
|
||||
|
||||
```
|
||||
stella export profile list --kind json --variant raw --format table
|
||||
```
|
||||
|
||||
Outputs columns `PROFILE`, `KIND`, `VARIANT`, `DISTRIBUTION`, `RETENTION`. Use `--format json` for automation.
|
||||
|
||||
### 2.2 `stella export profile show`
|
||||
|
||||
```
|
||||
stella export profile show prof-json-raw --output profile.json
|
||||
```
|
||||
|
||||
Fetches full configuration and writes it to file.
|
||||
|
||||
### 2.3 `stella export profile create`
|
||||
|
||||
```
|
||||
stella export profile create --file profiles/prof-json-raw.json
|
||||
```
|
||||
|
||||
JSON schema matches `POST /api/export/profiles`. CLI validates against built-in schema before submission. Requires `export:profile:manage`.
|
||||
|
||||
### 2.4 `stella export profile update`
|
||||
|
||||
```
|
||||
stella export profile update prof-json-raw \
|
||||
--retention "days:21" \
|
||||
--distribution http,object
|
||||
```
|
||||
|
||||
Supports toggling retention, adding/removing distribution targets, and renaming. Structural changes (kind, variant, include set) require editing the JSON and using `--replace-file` to create a new revision.
|
||||
|
||||
### 2.5 `stella export profile archive`
|
||||
|
||||
```
|
||||
stella export profile archive prof-json-raw --reason "Superseded by Phase 2 profile"
|
||||
```
|
||||
|
||||
Marks the profile inactive. Use `stella export profile restore` to re-activate.
|
||||
|
||||
## 3. Run lifecycle commands
|
||||
|
||||
### 3.1 `stella export run submit`
|
||||
|
||||
```
|
||||
stella export run submit prof-json-raw \
|
||||
--selector tenant=acme \
|
||||
--selector product=registry.example.com/app:* \
|
||||
--selector time=2025-10-01T00:00:00Z,2025-10-29T00:00:00Z \
|
||||
--policy-snapshot policy-snap-42 \
|
||||
--allow-empty=false
|
||||
```
|
||||
|
||||
Selectors accept `key=value` pairs; use `time=<from>,<to>` for windows. The command prints the `runId` and initial status.
|
||||
|
||||
### 3.2 `stella export run ls`
|
||||
|
||||
```
|
||||
stella export run ls --profile prof-json-raw --status active --tail 5
|
||||
```
|
||||
|
||||
Shows recent runs with columns `RUN`, `PROFILE`, `STATUS`, `PROGRESS`, `UPDATED`.
|
||||
|
||||
### 3.3 `stella export run show`
|
||||
|
||||
```
|
||||
stella export run show run-20251029-01 --format json
|
||||
```
|
||||
|
||||
Outputs full metadata, progress counters, distribution descriptors, and links.
|
||||
|
||||
### 3.4 `stella export run watch`
|
||||
|
||||
```
|
||||
stella export run watch run-20251029-01 --follow
|
||||
```
|
||||
|
||||
Streams server-sent events and renders a live progress bar. `--json` prints raw events for scripting.
|
||||
|
||||
### 3.5 `stella export run cancel`
|
||||
|
||||
```
|
||||
stella export run cancel run-20251029-01 --reason "Replacing with refined selectors"
|
||||
```
|
||||
|
||||
Gracefully cancels the run; exit code `0` indicates cancellation request accepted.
|
||||
|
||||
## 4. Download and verification commands
|
||||
|
||||
### 4.1 `stella export download`
|
||||
|
||||
```
|
||||
stella export download run-20251029-01 \
|
||||
--output out/exports/run-20251029-01.tar.zst \
|
||||
--resume
|
||||
```
|
||||
|
||||
Downloads the primary bundle. `--resume` enables HTTP range requests; the CLI checkpoints progress to `.part` files.
|
||||
|
||||
### 4.2 `stella export manifest`
|
||||
|
||||
```
|
||||
stella export manifest run-20251029-01 --output manifests/export.json
|
||||
```
|
||||
|
||||
Fetches the signed manifest. Use `--signature manifests/export.json.sig` to save the detached signature.
|
||||
|
||||
### 4.3 `stella export provenance`
|
||||
|
||||
```
|
||||
stella export provenance run-20251029-01 --output manifests/provenance.json
|
||||
```
|
||||
|
||||
Retrieves the signed provenance file. `--signature` behaves like the manifest command.
|
||||
|
||||
### 4.4 `stella export verify`
|
||||
|
||||
```
|
||||
stella export verify run-20251029-01 \
|
||||
--manifest manifests/export.json \
|
||||
--provenance manifests/provenance.json \
|
||||
--key keys/acme-export.pub
|
||||
```
|
||||
|
||||
Wrapper around `cosign verify`. Returns exit `0` when signatures and digests validate. Exit `20` when verification fails.
|
||||
|
||||
## 5. CI recipe (GitHub Actions example)
|
||||
|
||||
```yaml
|
||||
name: Export Center Bundle
|
||||
on:
|
||||
workflow_dispatch:
|
||||
jobs:
|
||||
export:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
- name: Install Stella CLI
|
||||
run: curl -sSfL https://downloads.stellaops.org/cli/install.sh | sh
|
||||
- name: Submit export run
|
||||
env:
|
||||
STELLA_TOKEN: ${{ secrets.STELLA_TOKEN }}
|
||||
run: |
|
||||
run_id=$(stella export run submit prof-json-raw \
|
||||
--selector tenant=acme \
|
||||
--selector product=registry.example.com/app:* \
|
||||
--allow-empty=false \
|
||||
--format json | jq -r '.runId')
|
||||
echo "RUN_ID=$run_id" >> $GITHUB_ENV
|
||||
- name: Wait for completion
|
||||
env:
|
||||
STELLA_TOKEN: ${{ secrets.STELLA_TOKEN }}
|
||||
run: |
|
||||
stella export run watch "$RUN_ID" --json \
|
||||
| tee artifacts/run.log \
|
||||
| jq -e 'select(.event == "run.succeeded")' > /dev/null
|
||||
- name: Download bundle
|
||||
env:
|
||||
STELLA_TOKEN: ${{ secrets.STELLA_TOKEN }}
|
||||
run: |
|
||||
stella export download "$RUN_ID" --output artifacts/export.tar.zst --resume
|
||||
stella export manifest "$RUN_ID" --output artifacts/export.json --signature artifacts/export.json.sig
|
||||
stella export provenance "$RUN_ID" --output artifacts/provenance.json --signature artifacts/provenance.json.sig
|
||||
- name: Verify signatures
|
||||
run: |
|
||||
stella export verify "$RUN_ID" \
|
||||
--manifest artifacts/export.json \
|
||||
--provenance artifacts/provenance.json \
|
||||
--key keys/acme-export.pub
|
||||
```
|
||||
|
||||
## 6. Exit codes
|
||||
|
||||
| Code | Meaning |
|
||||
|------|---------|
|
||||
| `0` | Command succeeded. |
|
||||
| `10` | Validation error (`ERR_EXPORT_001`). |
|
||||
| `11` | Profile missing or inaccessible (`ERR_EXPORT_002`). |
|
||||
| `12` | Quota or concurrency exceeded (`ERR_EXPORT_003` or `ERR_EXPORT_QUOTA`). |
|
||||
| `13` | Run failed due to adapter/signing/distribution error. |
|
||||
| `20` | Verification failure (`stella export verify`). |
|
||||
| `21` | Download incomplete after retries (network errors). |
|
||||
| `30` | CLI configuration error (missing token, invalid profile file). |
|
||||
|
||||
Exit codes above 100 are reserved for future profile-specific tooling.
|
||||
|
||||
## 7. Offline usage notes
|
||||
|
||||
- Use profiles that enable `object` distribution with local object storage endpoints. CLI reads `STELLA_EXPORT_OBJECT_ENDPOINT` when provided (falls back to gateway).
|
||||
- Mirror bundles work offline by skipping OCI distribution. CLI adds `--offline` to bypass OCI checks.
|
||||
- `stella export verify` works fully offline when provided with tenant public keys (packaged in Offline Kit).
|
||||
|
||||
## 8. Related documentation
|
||||
|
||||
- [Export Center Profiles](profiles.md)
|
||||
- [Export Center API reference](api.md)
|
||||
- [Export Center Architecture](architecture.md)
|
||||
- [Aggregation-Only Contract reference](../ingestion/aggregation-only-contract.md)
|
||||
|
||||
> **Imposed rule:** Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.
|
||||
# Stella CLI - Export Center Commands
|
||||
|
||||
> **Audience:** Operators, release engineers, and CI maintainers using the `stella` CLI to manage Export Center profiles and runs.
|
||||
> **Supported from:** `stella` CLI >= 0.22.0 (Export Center Phase 1).
|
||||
> **Prerequisites:** Authority token with the scopes noted per command (`export:profile:manage`, `export:run`, `export:read`, `export:download`).
|
||||
|
||||
Use this guide with the [Export Center API reference](api.md) and [Profiles](profiles.md) catalogue. The CLI wraps the same REST endpoints, preserving deterministic behaviour and guardrails.
|
||||
|
||||
> Status: CLI support is tracked under `CLI-EXPORT-35-001` and `CLI-EXPORT-36-001`. The current CLI build does not yet surface these commands; treat this guide as the target contract and adjust once implementations merge.
|
||||
|
||||
## 1. Global options and configuration
|
||||
|
||||
| Flag | Default | Description |
|
||||
|------|---------|-------------|
|
||||
| `--server <url>` | `https://stella.local` | Gateway root. Matches `STELLA_SERVER`. |
|
||||
| `--tenant <id>` | Token tenant | Override tenant for multi-tenant tokens. |
|
||||
| `--profile <name>` | none | Loads saved defaults from `~/.stella/profiles/<name>.toml`. |
|
||||
| `--output <file>` | stdout | Redirect full JSON response. |
|
||||
| `--format <table|json|yaml>` | `table` on TTY | Controls table formatting for list commands. |
|
||||
| `--trace` | false | Emit request timing and correlation ids. |
|
||||
|
||||
Environment variables: `STELLA_TOKEN`, `STELLA_SERVER`, `STELLA_TENANT`, `STELLA_PROFILE`.
|
||||
|
||||
Exit codes align with API error codes (see section 6).
|
||||
|
||||
## 2. Profile management commands
|
||||
|
||||
### 2.1 `stella export profile list`
|
||||
|
||||
List profiles for the current tenant.
|
||||
|
||||
```
|
||||
stella export profile list --kind json --variant raw --format table
|
||||
```
|
||||
|
||||
Outputs columns `PROFILE`, `KIND`, `VARIANT`, `DISTRIBUTION`, `RETENTION`. Use `--format json` for automation.
|
||||
|
||||
### 2.2 `stella export profile show`
|
||||
|
||||
```
|
||||
stella export profile show prof-json-raw --output profile.json
|
||||
```
|
||||
|
||||
Fetches full configuration and writes it to file.
|
||||
|
||||
### 2.3 `stella export profile create`
|
||||
|
||||
```
|
||||
stella export profile create --file profiles/prof-json-raw.json
|
||||
```
|
||||
|
||||
JSON schema matches `POST /api/export/profiles`. CLI validates against built-in schema before submission. Requires `export:profile:manage`.
|
||||
|
||||
### 2.4 `stella export profile update`
|
||||
|
||||
```
|
||||
stella export profile update prof-json-raw \
|
||||
--retention "days:21" \
|
||||
--distribution http,object
|
||||
```
|
||||
|
||||
Supports toggling retention, adding/removing distribution targets, and renaming. Structural changes (kind, variant, include set) require editing the JSON and using `--replace-file` to create a new revision.
|
||||
|
||||
### 2.5 `stella export profile archive`
|
||||
|
||||
```
|
||||
stella export profile archive prof-json-raw --reason "Superseded by Phase 2 profile"
|
||||
```
|
||||
|
||||
Marks the profile inactive. Use `stella export profile restore` to re-activate.
|
||||
|
||||
## 3. Run lifecycle commands
|
||||
|
||||
### 3.1 `stella export run submit`
|
||||
|
||||
```
|
||||
stella export run submit prof-json-raw \
|
||||
--selector tenant=acme \
|
||||
--selector product=registry.example.com/app:* \
|
||||
--selector time=2025-10-01T00:00:00Z,2025-10-29T00:00:00Z \
|
||||
--policy-snapshot policy-snap-42 \
|
||||
--allow-empty=false
|
||||
```
|
||||
|
||||
Selectors accept `key=value` pairs; use `time=<from>,<to>` for windows. The command prints the `runId` and initial status.
|
||||
|
||||
### 3.2 `stella export run ls`
|
||||
|
||||
```
|
||||
stella export run ls --profile prof-json-raw --status active --tail 5
|
||||
```
|
||||
|
||||
Shows recent runs with columns `RUN`, `PROFILE`, `STATUS`, `PROGRESS`, `UPDATED`.
|
||||
|
||||
### 3.3 `stella export run show`
|
||||
|
||||
```
|
||||
stella export run show run-20251029-01 --format json
|
||||
```
|
||||
|
||||
Outputs full metadata, progress counters, distribution descriptors, and links.
|
||||
|
||||
### 3.4 `stella export run watch`
|
||||
|
||||
```
|
||||
stella export run watch run-20251029-01 --follow
|
||||
```
|
||||
|
||||
Streams server-sent events and renders a live progress bar. `--json` prints raw events for scripting.
|
||||
|
||||
### 3.5 `stella export run cancel`
|
||||
|
||||
```
|
||||
stella export run cancel run-20251029-01 --reason "Replacing with refined selectors"
|
||||
```
|
||||
|
||||
Gracefully cancels the run; exit code `0` indicates cancellation request accepted.
|
||||
|
||||
## 4. Download and verification commands
|
||||
|
||||
### 4.1 `stella export download`
|
||||
|
||||
```
|
||||
stella export download run-20251029-01 \
|
||||
--output out/exports/run-20251029-01.tar.zst \
|
||||
--resume
|
||||
```
|
||||
|
||||
Downloads the primary bundle. `--resume` enables HTTP range requests; the CLI checkpoints progress to `.part` files.
|
||||
|
||||
### 4.2 `stella export manifest`
|
||||
|
||||
```
|
||||
stella export manifest run-20251029-01 --output manifests/export.json
|
||||
```
|
||||
|
||||
Fetches the signed manifest. Use `--signature manifests/export.json.sig` to save the detached signature.
|
||||
|
||||
### 4.3 `stella export provenance`
|
||||
|
||||
```
|
||||
stella export provenance run-20251029-01 --output manifests/provenance.json
|
||||
```
|
||||
|
||||
Retrieves the signed provenance file. `--signature` behaves like the manifest command.
|
||||
|
||||
### 4.4 `stella export verify`
|
||||
|
||||
```
|
||||
stella export verify run-20251029-01 \
|
||||
--manifest manifests/export.json \
|
||||
--provenance manifests/provenance.json \
|
||||
--key keys/acme-export.pub
|
||||
```
|
||||
|
||||
Wrapper around `cosign verify`. Returns exit `0` when signatures and digests validate. Exit `20` when verification fails.
|
||||
|
||||
## 5. CI recipe (GitHub Actions example)
|
||||
|
||||
```yaml
|
||||
name: Export Center Bundle
|
||||
on:
|
||||
workflow_dispatch:
|
||||
jobs:
|
||||
export:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
- name: Install Stella CLI
|
||||
run: curl -sSfL https://downloads.stellaops.org/cli/install.sh | sh
|
||||
- name: Submit export run
|
||||
env:
|
||||
STELLA_TOKEN: ${{ secrets.STELLA_TOKEN }}
|
||||
run: |
|
||||
run_id=$(stella export run submit prof-json-raw \
|
||||
--selector tenant=acme \
|
||||
--selector product=registry.example.com/app:* \
|
||||
--allow-empty=false \
|
||||
--format json | jq -r '.runId')
|
||||
echo "RUN_ID=$run_id" >> $GITHUB_ENV
|
||||
- name: Wait for completion
|
||||
env:
|
||||
STELLA_TOKEN: ${{ secrets.STELLA_TOKEN }}
|
||||
run: |
|
||||
stella export run watch "$RUN_ID" --json \
|
||||
| tee artifacts/run.log \
|
||||
| jq -e 'select(.event == "run.succeeded")' > /dev/null
|
||||
- name: Download bundle
|
||||
env:
|
||||
STELLA_TOKEN: ${{ secrets.STELLA_TOKEN }}
|
||||
run: |
|
||||
stella export download "$RUN_ID" --output artifacts/export.tar.zst --resume
|
||||
stella export manifest "$RUN_ID" --output artifacts/export.json --signature artifacts/export.json.sig
|
||||
stella export provenance "$RUN_ID" --output artifacts/provenance.json --signature artifacts/provenance.json.sig
|
||||
- name: Verify signatures
|
||||
run: |
|
||||
stella export verify "$RUN_ID" \
|
||||
--manifest artifacts/export.json \
|
||||
--provenance artifacts/provenance.json \
|
||||
--key keys/acme-export.pub
|
||||
```
|
||||
|
||||
## 6. Exit codes
|
||||
|
||||
| Code | Meaning |
|
||||
|------|---------|
|
||||
| `0` | Command succeeded. |
|
||||
| `10` | Validation error (`ERR_EXPORT_001`). |
|
||||
| `11` | Profile missing or inaccessible (`ERR_EXPORT_002`). |
|
||||
| `12` | Quota or concurrency exceeded (`ERR_EXPORT_003` or `ERR_EXPORT_QUOTA`). |
|
||||
| `13` | Run failed due to adapter/signing/distribution error. |
|
||||
| `20` | Verification failure (`stella export verify`). |
|
||||
| `21` | Download incomplete after retries (network errors). |
|
||||
| `30` | CLI configuration error (missing token, invalid profile file). |
|
||||
|
||||
Exit codes above 100 are reserved for future profile-specific tooling.
|
||||
|
||||
## 7. Offline usage notes
|
||||
|
||||
- Use profiles that enable `object` distribution with local object storage endpoints. CLI reads `STELLA_EXPORT_OBJECT_ENDPOINT` when provided (falls back to gateway).
|
||||
- Mirror bundles work offline by skipping OCI distribution. CLI adds `--offline` to bypass OCI checks.
|
||||
- `stella export verify` works fully offline when provided with tenant public keys (packaged in Offline Kit).
|
||||
|
||||
## 8. Related documentation
|
||||
|
||||
- [Export Center Profiles](profiles.md)
|
||||
- [Export Center API reference](api.md)
|
||||
- [Export Center Architecture](architecture.md)
|
||||
- [Aggregation-Only Contract reference](../ingestion/aggregation-only-contract.md)
|
||||
|
||||
> **Imposed rule:** Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.
|
||||
|
||||
@@ -1,202 +1,202 @@
|
||||
# Export Center Mirror Bundles
|
||||
|
||||
Mirror bundles package StellaOps evidence, policy overlays, and indexes for air-gapped or bandwidth-constrained environments. They are produced by the `mirror:full` and `mirror:delta` profiles described in Epic 10 (Export Center) and implemented across Sprints 35-37 (`EXPORT-SVC-35-004`, `EXPORT-SVC-37-001`, `EXPORT-SVC-37-002`). This guide details bundle layouts, delta mechanics, encryption workflow, import procedures, and operational best practices.
|
||||
|
||||
> Export Center workers are being wired while this document is written. Treat the content as the target contract for adapter development and update specifics as the implementation lands.
|
||||
|
||||
## 1. Bundle overview
|
||||
|
||||
| Profile | Contents | Typical use cases | Distribution |
|
||||
|---------|----------|-------------------|--------------|
|
||||
| `mirror:full` | Complete snapshot of raw evidence, normalized records, indexes, policy snapshots, provenance, signatures. | Initial seeding of an air-gapped mirror, disaster recovery drills. | Download bundle, optional OCI artifact. |
|
||||
| `mirror:delta` | Changes since a specified base export: added/updated/removed advisories, VEX statements, SBOMs, indexes, manifests. | Incremental updates, bandwidth reduction, nightly refreshes. | Download bundle, optional OCI artifact. |
|
||||
|
||||
Both profiles respect AOC boundaries: raw ingestion data remains untouched, and policy outputs live under their own directory with explicit provenance.
|
||||
|
||||
## 2. Filesystem layout
|
||||
|
||||
Directory structure inside the extracted bundle:
|
||||
|
||||
```
|
||||
mirror/
|
||||
manifest.yaml
|
||||
export.json
|
||||
provenance.json
|
||||
README.md
|
||||
indexes/
|
||||
advisories.index.json
|
||||
vex.index.json
|
||||
sbom.index.json
|
||||
findings.index.json
|
||||
data/
|
||||
raw/
|
||||
advisories/*.jsonl.zst
|
||||
vex/*.jsonl.zst
|
||||
sboms/<subject>/sbom.json
|
||||
normalized/
|
||||
advisories/*.jsonl.zst
|
||||
vex/*.jsonl.zst
|
||||
policy/
|
||||
snapshot.json
|
||||
evaluations.jsonl.zst
|
||||
consensus/
|
||||
vex_consensus.jsonl.zst
|
||||
signatures/
|
||||
export.sig
|
||||
manifest.sig
|
||||
```
|
||||
|
||||
`manifest.yaml` summarises profile metadata, selectors, counts, sizes, and SHA-256 digests. `export.json` and `provenance.json` mirror the JSON profile manifests and are signed using the configured KMS key.
|
||||
|
||||
Example `manifest.yaml`:
|
||||
|
||||
```yaml
|
||||
profile: mirror:full
|
||||
runId: run-20251029-01
|
||||
tenant: acme
|
||||
selectors:
|
||||
products:
|
||||
- registry.example.com/app:*
|
||||
timeWindow:
|
||||
from: 2025-10-01T00:00:00Z
|
||||
to: 2025-10-29T00:00:00Z
|
||||
counts:
|
||||
advisories: 15234
|
||||
vex: 3045
|
||||
sboms: 872
|
||||
policyEvaluations: 19876
|
||||
artifacts:
|
||||
- path: data/raw/advisories/a0.jsonl.zst
|
||||
sha256: 9f4b...
|
||||
bytes: 7340021
|
||||
encryption:
|
||||
mode: age
|
||||
strict: false
|
||||
recipients:
|
||||
- age1tenantkey...
|
||||
```
|
||||
|
||||
## 3. Delta mechanics
|
||||
|
||||
Delta bundles reference a previous full or delta run via `baseExportId` and `baseManifestDigest`. They contain:
|
||||
|
||||
```
|
||||
delta/
|
||||
changed/
|
||||
data/raw/advisories/*.jsonl.zst
|
||||
...
|
||||
removed/
|
||||
advisories.jsonl # list of advisory IDs removed
|
||||
vex.jsonl
|
||||
sboms.jsonl
|
||||
manifest.diff.json # summary of counts, hashes, base export metadata
|
||||
```
|
||||
|
||||
- **Base lookup:** The worker verifies that the base export is reachable (download path or OCI reference). If missing, the run fails with `ERR_EXPORT_BASE_MISSING`.
|
||||
- **Change detection:** Uses deterministic hashing of normalized records to compute additions/updates. Indexes are regenerated only for affected subjects.
|
||||
- **Application order:** Consumers apply deltas sequentially. A `resetBaseline=true` flag instructs them to drop cached state and apply the bundle as a full refresh.
|
||||
|
||||
Example `manifest.diff.json` (delta):
|
||||
|
||||
```json
|
||||
{
|
||||
"baseExportId": "run-20251025-01",
|
||||
"baseManifestDigest": "sha256:aa11...",
|
||||
"resetBaseline": false,
|
||||
"added": {
|
||||
"advisories": 43,
|
||||
"vex": 12,
|
||||
"sboms": 5
|
||||
},
|
||||
"changed": {
|
||||
"advisories": 18,
|
||||
"vex": 7
|
||||
},
|
||||
"removed": {
|
||||
"advisories": 2,
|
||||
"vex": 0,
|
||||
"sboms": 0
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## 4. Encryption workflow
|
||||
|
||||
Mirror bundles support optional encryption of the `data/` subtree:
|
||||
|
||||
- **Algorithm:** Age (X25519) or AES-GCM (256-bit) based on profile configuration.
|
||||
- **Key wrapping:** Keys fetched from Authority-managed KMS through Export Center. Wrapped data keys stored in `provenance.json` under `encryption.recipients[]`.
|
||||
- **Metadata:** `manifest.yaml` records `encryption.mode`, `recipients`, and `encryptedPaths`.
|
||||
- **Strict mode:** `strict=true` encrypts everything except `manifest.yaml` and `export.json`. Default (`false`) leaves manifests unencrypted to simplify discovery.
|
||||
- **Verification:** CLI (`stella export verify`) and Offline Kit scripts perform signature checks prior to decryption.
|
||||
|
||||
Operators must distribute recipient keys out of band. Export Center does not transmit private keys.
|
||||
|
||||
## 5. Import workflow
|
||||
|
||||
### 5.1 Offline Kit
|
||||
|
||||
Offline Kit bundles reference the latest full mirror export plus the last `N` deltas. Administrators run:
|
||||
|
||||
```
|
||||
./offline-kit/bin/mirror import /path/to/mirror-20251029-full.tar.zst
|
||||
./offline-kit/bin/mirror import /path/to/mirror-20251030-delta.tar.zst
|
||||
```
|
||||
|
||||
The tool verifies signatures, applies deltas, and updates the mirror index served by the local gateway.
|
||||
|
||||
### 5.2 Custom automation
|
||||
|
||||
1. Download bundle (`stella export download`) and verify signatures (`stella export verify`).
|
||||
2. Extract archive into a staging directory.
|
||||
3. For encrypted bundles, decrypt using the provided age/AES key.
|
||||
4. Sync `mirror/data` onto the target mirror store (object storage, NFS, etc.).
|
||||
5. Republish indexes or reload services that depend on the mirror.
|
||||
|
||||
Delta consumers must track `appliedExportIds` to ensure ordering.
|
||||
|
||||
Sequence diagram of download/import:
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant CLI as stella CLI
|
||||
participant Mirror as Mirror Store
|
||||
participant Verify as Verification Tool
|
||||
CLI->>CLI: stella export download run-20251029-01
|
||||
CLI->>Verify: stella export verify run-20251029-01
|
||||
CLI->>Mirror: mirror import mirror-20251029-full.tar.zst
|
||||
CLI->>Mirror: mirror import mirror-20251030-delta.tar.zst
|
||||
Mirror-->>CLI: import complete (run-20251030-02)
|
||||
```
|
||||
|
||||
## 6. Operational guidance
|
||||
|
||||
- **Retention:** Keep at least one full bundle plus the deltas required for disaster recovery. Configure `ExportCenter:Retention:Mirror` to prune older bundles automatically.
|
||||
- **Storage footprint:** Full bundles can exceed tens of gigabytes. Plan object storage or NAS capacity accordingly and enable compression (`compression.codec=zstd`).
|
||||
- **Scheduling:** For high-churn environments, run daily full exports and hourly deltas. Record cadence in `manifest.yaml` (`schedule.cron`).
|
||||
- **Incident recovery:** To rebuild a mirror:
|
||||
1. Apply the most recent full bundle.
|
||||
2. Apply deltas in order of `createdAt`.
|
||||
3. Re-run integrity checks (`mirror verify <path>`).
|
||||
- **Audit logging:** Export Center logs `mirror.bundle.created`, `mirror.delta.applied`, and `mirror.encryption.enabled` events. Consume them in the central observability pipeline.
|
||||
|
||||
## 7. Troubleshooting
|
||||
|
||||
| Symptom | Meaning | Action |
|
||||
|---------|---------|--------|
|
||||
| `ERR_EXPORT_BASE_MISSING` | Base export not available | Republish base bundle or rebuild as full export. |
|
||||
| Delta applies but mirror misses entries | Deltas applied out of order | Rebuild from last full bundle and reapply in sequence. |
|
||||
| Decryption fails | Recipient key mismatch or corrupted bundle | Confirm key distribution and re-download bundle. |
|
||||
| Verification errors | Signature mismatch | Do not import; regenerate bundle and investigate signing pipeline. |
|
||||
| Manifest hash mismatch | Files changed after extraction | Re-extract bundle and re-run verification; check storage tampering. |
|
||||
|
||||
## 8. References
|
||||
|
||||
- [Export Center Overview](overview.md)
|
||||
- [Export Center Architecture](architecture.md)
|
||||
- [Export Center API reference](api.md)
|
||||
- [Export Center CLI Guide](cli.md)
|
||||
- [Concelier mirror runbook](../ops/concelier-mirror-operations.md)
|
||||
- [Aggregation-Only Contract reference](../ingestion/aggregation-only-contract.md)
|
||||
|
||||
> **Imposed rule:** Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.
|
||||
# Export Center Mirror Bundles
|
||||
|
||||
Mirror bundles package StellaOps evidence, policy overlays, and indexes for air-gapped or bandwidth-constrained environments. They are produced by the `mirror:full` and `mirror:delta` profiles described in Epic 10 (Export Center) and implemented across Sprints 35-37 (`EXPORT-SVC-35-004`, `EXPORT-SVC-37-001`, `EXPORT-SVC-37-002`). This guide details bundle layouts, delta mechanics, encryption workflow, import procedures, and operational best practices.
|
||||
|
||||
> Export Center workers are being wired while this document is written. Treat the content as the target contract for adapter development and update specifics as the implementation lands.
|
||||
|
||||
## 1. Bundle overview
|
||||
|
||||
| Profile | Contents | Typical use cases | Distribution |
|
||||
|---------|----------|-------------------|--------------|
|
||||
| `mirror:full` | Complete snapshot of raw evidence, normalized records, indexes, policy snapshots, provenance, signatures. | Initial seeding of an air-gapped mirror, disaster recovery drills. | Download bundle, optional OCI artifact. |
|
||||
| `mirror:delta` | Changes since a specified base export: added/updated/removed advisories, VEX statements, SBOMs, indexes, manifests. | Incremental updates, bandwidth reduction, nightly refreshes. | Download bundle, optional OCI artifact. |
|
||||
|
||||
Both profiles respect AOC boundaries: raw ingestion data remains untouched, and policy outputs live under their own directory with explicit provenance.
|
||||
|
||||
## 2. Filesystem layout
|
||||
|
||||
Directory structure inside the extracted bundle:
|
||||
|
||||
```
|
||||
mirror/
|
||||
manifest.yaml
|
||||
export.json
|
||||
provenance.json
|
||||
README.md
|
||||
indexes/
|
||||
advisories.index.json
|
||||
vex.index.json
|
||||
sbom.index.json
|
||||
findings.index.json
|
||||
data/
|
||||
raw/
|
||||
advisories/*.jsonl.zst
|
||||
vex/*.jsonl.zst
|
||||
sboms/<subject>/sbom.json
|
||||
normalized/
|
||||
advisories/*.jsonl.zst
|
||||
vex/*.jsonl.zst
|
||||
policy/
|
||||
snapshot.json
|
||||
evaluations.jsonl.zst
|
||||
consensus/
|
||||
vex_consensus.jsonl.zst
|
||||
signatures/
|
||||
export.sig
|
||||
manifest.sig
|
||||
```
|
||||
|
||||
`manifest.yaml` summarises profile metadata, selectors, counts, sizes, and SHA-256 digests. `export.json` and `provenance.json` mirror the JSON profile manifests and are signed using the configured KMS key.
|
||||
|
||||
Example `manifest.yaml`:
|
||||
|
||||
```yaml
|
||||
profile: mirror:full
|
||||
runId: run-20251029-01
|
||||
tenant: acme
|
||||
selectors:
|
||||
products:
|
||||
- registry.example.com/app:*
|
||||
timeWindow:
|
||||
from: 2025-10-01T00:00:00Z
|
||||
to: 2025-10-29T00:00:00Z
|
||||
counts:
|
||||
advisories: 15234
|
||||
vex: 3045
|
||||
sboms: 872
|
||||
policyEvaluations: 19876
|
||||
artifacts:
|
||||
- path: data/raw/advisories/a0.jsonl.zst
|
||||
sha256: 9f4b...
|
||||
bytes: 7340021
|
||||
encryption:
|
||||
mode: age
|
||||
strict: false
|
||||
recipients:
|
||||
- age1tenantkey...
|
||||
```
|
||||
|
||||
## 3. Delta mechanics
|
||||
|
||||
Delta bundles reference a previous full or delta run via `baseExportId` and `baseManifestDigest`. They contain:
|
||||
|
||||
```
|
||||
delta/
|
||||
changed/
|
||||
data/raw/advisories/*.jsonl.zst
|
||||
...
|
||||
removed/
|
||||
advisories.jsonl # list of advisory IDs removed
|
||||
vex.jsonl
|
||||
sboms.jsonl
|
||||
manifest.diff.json # summary of counts, hashes, base export metadata
|
||||
```
|
||||
|
||||
- **Base lookup:** The worker verifies that the base export is reachable (download path or OCI reference). If missing, the run fails with `ERR_EXPORT_BASE_MISSING`.
|
||||
- **Change detection:** Uses deterministic hashing of normalized records to compute additions/updates. Indexes are regenerated only for affected subjects.
|
||||
- **Application order:** Consumers apply deltas sequentially. A `resetBaseline=true` flag instructs them to drop cached state and apply the bundle as a full refresh.
|
||||
|
||||
Example `manifest.diff.json` (delta):
|
||||
|
||||
```json
|
||||
{
|
||||
"baseExportId": "run-20251025-01",
|
||||
"baseManifestDigest": "sha256:aa11...",
|
||||
"resetBaseline": false,
|
||||
"added": {
|
||||
"advisories": 43,
|
||||
"vex": 12,
|
||||
"sboms": 5
|
||||
},
|
||||
"changed": {
|
||||
"advisories": 18,
|
||||
"vex": 7
|
||||
},
|
||||
"removed": {
|
||||
"advisories": 2,
|
||||
"vex": 0,
|
||||
"sboms": 0
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## 4. Encryption workflow
|
||||
|
||||
Mirror bundles support optional encryption of the `data/` subtree:
|
||||
|
||||
- **Algorithm:** Age (X25519) or AES-GCM (256-bit) based on profile configuration.
|
||||
- **Key wrapping:** Keys fetched from Authority-managed KMS through Export Center. Wrapped data keys stored in `provenance.json` under `encryption.recipients[]`.
|
||||
- **Metadata:** `manifest.yaml` records `encryption.mode`, `recipients`, and `encryptedPaths`.
|
||||
- **Strict mode:** `strict=true` encrypts everything except `manifest.yaml` and `export.json`. Default (`false`) leaves manifests unencrypted to simplify discovery.
|
||||
- **Verification:** CLI (`stella export verify`) and Offline Kit scripts perform signature checks prior to decryption.
|
||||
|
||||
Operators must distribute recipient keys out of band. Export Center does not transmit private keys.
|
||||
|
||||
## 5. Import workflow
|
||||
|
||||
### 5.1 Offline Kit
|
||||
|
||||
Offline Kit bundles reference the latest full mirror export plus the last `N` deltas. Administrators run:
|
||||
|
||||
```
|
||||
./offline-kit/bin/mirror import /path/to/mirror-20251029-full.tar.zst
|
||||
./offline-kit/bin/mirror import /path/to/mirror-20251030-delta.tar.zst
|
||||
```
|
||||
|
||||
The tool verifies signatures, applies deltas, and updates the mirror index served by the local gateway.
|
||||
|
||||
### 5.2 Custom automation
|
||||
|
||||
1. Download bundle (`stella export download`) and verify signatures (`stella export verify`).
|
||||
2. Extract archive into a staging directory.
|
||||
3. For encrypted bundles, decrypt using the provided age/AES key.
|
||||
4. Sync `mirror/data` onto the target mirror store (object storage, NFS, etc.).
|
||||
5. Republish indexes or reload services that depend on the mirror.
|
||||
|
||||
Delta consumers must track `appliedExportIds` to ensure ordering.
|
||||
|
||||
Sequence diagram of download/import:
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant CLI as stella CLI
|
||||
participant Mirror as Mirror Store
|
||||
participant Verify as Verification Tool
|
||||
CLI->>CLI: stella export download run-20251029-01
|
||||
CLI->>Verify: stella export verify run-20251029-01
|
||||
CLI->>Mirror: mirror import mirror-20251029-full.tar.zst
|
||||
CLI->>Mirror: mirror import mirror-20251030-delta.tar.zst
|
||||
Mirror-->>CLI: import complete (run-20251030-02)
|
||||
```
|
||||
|
||||
## 6. Operational guidance
|
||||
|
||||
- **Retention:** Keep at least one full bundle plus the deltas required for disaster recovery. Configure `ExportCenter:Retention:Mirror` to prune older bundles automatically.
|
||||
- **Storage footprint:** Full bundles can exceed tens of gigabytes. Plan object storage or NAS capacity accordingly and enable compression (`compression.codec=zstd`).
|
||||
- **Scheduling:** For high-churn environments, run daily full exports and hourly deltas. Record cadence in `manifest.yaml` (`schedule.cron`).
|
||||
- **Incident recovery:** To rebuild a mirror:
|
||||
1. Apply the most recent full bundle.
|
||||
2. Apply deltas in order of `createdAt`.
|
||||
3. Re-run integrity checks (`mirror verify <path>`).
|
||||
- **Audit logging:** Export Center logs `mirror.bundle.created`, `mirror.delta.applied`, and `mirror.encryption.enabled` events. Consume them in the central observability pipeline.
|
||||
|
||||
## 7. Troubleshooting
|
||||
|
||||
| Symptom | Meaning | Action |
|
||||
|---------|---------|--------|
|
||||
| `ERR_EXPORT_BASE_MISSING` | Base export not available | Republish base bundle or rebuild as full export. |
|
||||
| Delta applies but mirror misses entries | Deltas applied out of order | Rebuild from last full bundle and reapply in sequence. |
|
||||
| Decryption fails | Recipient key mismatch or corrupted bundle | Confirm key distribution and re-download bundle. |
|
||||
| Verification errors | Signature mismatch | Do not import; regenerate bundle and investigate signing pipeline. |
|
||||
| Manifest hash mismatch | Files changed after extraction | Re-extract bundle and re-run verification; check storage tampering. |
|
||||
|
||||
## 8. References
|
||||
|
||||
- [Export Center Overview](overview.md)
|
||||
- [Export Center Architecture](architecture.md)
|
||||
- [Export Center API reference](api.md)
|
||||
- [Export Center CLI Guide](cli.md)
|
||||
- [Concelier mirror runbook](../ops/concelier-mirror-operations.md)
|
||||
- [Aggregation-Only Contract reference](../ingestion/aggregation-only-contract.md)
|
||||
|
||||
> **Imposed rule:** Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.
|
||||
|
||||
@@ -1,63 +1,63 @@
|
||||
# Export Center Overview
|
||||
|
||||
The Export Center packages StellaOps evidence and policy outputs into portable, verifiable bundles. It provides one workflow for operators to deliver advisories, SBOMs, VEX statements, and policy decisions into downstream systems or air-gapped environments without rewriting data or violating the Aggregation-Only Contract (AOC).
|
||||
|
||||
## What the Export Center delivers
|
||||
- **Unified export service.** A dedicated `exporter` service coordinates profiles, runs, signing, and distribution targets with deterministic manifests.
|
||||
- **Profile catalogue.** Out of the box variants include `json:raw`, `json:policy`, `trivy:db`, `trivy:java-db`, `mirror:full`, and `mirror:delta`, each aligned with AOC rules and downstream compatibility requirements.
|
||||
- **Surface parity.** Operators can create, monitor, and download exports through the Web API gateway, Console workflows, and the CLI (`stella export ...`). All surfaces enforce tenant scope and RBAC consistently.
|
||||
- **Automation hooks.** One-off, cron, and event triggers are orchestrated via the Scheduler/Orchestrator integration. Export telemetry (durations, bundle size, verification outcomes) feeds structured logs, metrics, and optional OpenTelemetry traces.
|
||||
|
||||
### Profile variants at a glance
|
||||
|
||||
| Profile | Contents | Primary scenarios | Distribution defaults |
|
||||
|---------|----------|-------------------|-----------------------|
|
||||
| `json:raw` | Canonical advisories, VEX, SBOM JSONL with hashes | Downstream analytics, evidence escrow | HTTP download, object storage |
|
||||
| `json:policy` | `json:raw` plus policy snapshot, evaluated findings | Policy attestation, audit packages | HTTP download, object storage |
|
||||
| `trivy:db` / `trivy:java-db` | Trivy-compatible vulnerability databases | Feed external scanners and CI | OCI artifact push, download |
|
||||
| `mirror:full` | Complete evidence, indexes, policy, provenance | Air-gap mirror, disaster recovery | Filesystem bundle, OCI artifact |
|
||||
| `mirror:delta` | Changes relative to prior manifest | Incremental updates to mirrors | Filesystem bundle, OCI artifact |
|
||||
|
||||
## How it works end-to-end
|
||||
1. **Profile & scope resolution.** A profile defines export type, content filters, and bundle settings. Scope selectors target tenants, artifacts, time windows, ecosystems, or SBOM subjects.
|
||||
2. **Ledger collection.** Workers stream canonical data from Findings Ledger, VEX Lens, Conseiller feeds, and SBOM service. Policy exports pin a deterministic policy snapshot from Policy Engine.
|
||||
3. **Adapter execution.** JSON adapters produce normalized `.jsonl.zst` outputs, Trivy adapters translate to the Trivy DB schema, and mirror adapters build filesystem or OCI bundle layouts.
|
||||
4. **Manifesting & provenance.** Every run emits `export.json` (profile, filters, counts, checksums) and `provenance.json` (source artifacts, policy snapshot ids, signature references).
|
||||
5. **Signing & distribution.** Bundles are signed via configured KMS (cosign-compatible) and distributed through HTTP streaming, OCI registry pushes, or object storage staging.
|
||||
|
||||
Refer to `docs/export-center/architecture.md` (Sprint 35 task) for component diagrams and adapter internals once published.
|
||||
|
||||
## Security and compliance guardrails
|
||||
- **AOC alignment.** Exports bundle raw evidence and optional policy evaluations without mutating source content. Policy overlays remain attributed to Policy Engine and are clearly partitioned.
|
||||
- **Tenant isolation.** All queries, manifests, and bundle paths carry tenant identifiers. Cross-tenant exports require explicit signed approval and ship with provenance trails.
|
||||
- **Signing and encryption.** Manifests and payloads are signed using the platform KMS. Mirror profiles support optional in-bundle encryption (age/AES-GCM) with key wrapping.
|
||||
- **Determinism.** Identical inputs yield identical bundles. Timestamps serialize in UTC ISO-8601; manifests include content hashes for audit replay.
|
||||
|
||||
See `docs/security/policy-governance.md` and `docs/ingestion/aggregation-only-contract.md` for broader guardrail context.
|
||||
|
||||
## Operating it offline
|
||||
- **Offline Kit integration.** Air-gapped deployments receive pre-built export profiles and object storage layout templates through the Offline Kit bundles.
|
||||
- **Mirror bundles.** `mirror:full` packages raw evidence, normalized indexes, policy snapshots, and provenance in a portable filesystem layout suitable for disconnected environments. `mirror:delta` tracks changes relative to a prior export manifest.
|
||||
- **No unsanctioned egress.** The exporter respects the platform allowlist. External calls (e.g., OCI pushes) require explicit configuration and are disabled by default for offline installs.
|
||||
|
||||
Consult `docs/24_OFFLINE_KIT.md` for Offline Kit delivery and `docs/ops/concelier-mirror-operations.md` for mirror ingestion procedures.
|
||||
|
||||
## Getting started
|
||||
1. **Choose a profile.** Map requirements to the profile table above. Policy-aware exports need a published policy snapshot.
|
||||
2. **Define selectors.** Decide on tenants, products, SBOM subjects, or time windows to include. Default selectors export the entire tenant scope.
|
||||
3. **Run via preferred surface.**
|
||||
- **Console:** Navigate to the Export Center view, create a run, monitor progress, and download artifacts.
|
||||
- **CLI:** Use `stella export run --profile <name> --selector <filters>` to submit a job, then `stella export download`.
|
||||
- **API:** POST to `/api/export/runs` with profile id and scope payload; stream results from `/api/export/runs/{id}/download`.
|
||||
4. **Verify bundles.** Use the attached provenance manifest and cosign signature to validate contents before distributing downstream.
|
||||
|
||||
Refer to `docs/export-center/cli.md` for detailed command syntax and automation examples.
|
||||
|
||||
## Observability & troubleshooting
|
||||
- Structured logs emit lifecycle events (`fetch`, `adapter`, `sign`, `publish`) with correlation IDs for parallel job tracing.
|
||||
- Metrics `exporter_run_duration_seconds`, `exporter_bundle_bytes_total`, and `exporter_run_failures_total` feed Grafana dashboards defined in the deployment runbooks.
|
||||
- Verification failures or schema mismatches bubble up through failure events and appear in Console/CLI with actionable error messages. Inspect the run's audit log and `provenance.json` for root cause.
|
||||
|
||||
See `docs/observability/policy.md` and `docs/ops/deployment-upgrade-runbook.md` for telemetry and operations guidance.
|
||||
|
||||
> **Imposed rule:** Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.
|
||||
# Export Center Overview
|
||||
|
||||
The Export Center packages StellaOps evidence and policy outputs into portable, verifiable bundles. It provides one workflow for operators to deliver advisories, SBOMs, VEX statements, and policy decisions into downstream systems or air-gapped environments without rewriting data or violating the Aggregation-Only Contract (AOC).
|
||||
|
||||
## What the Export Center delivers
|
||||
- **Unified export service.** A dedicated `exporter` service coordinates profiles, runs, signing, and distribution targets with deterministic manifests.
|
||||
- **Profile catalogue.** Out of the box variants include `json:raw`, `json:policy`, `trivy:db`, `trivy:java-db`, `mirror:full`, and `mirror:delta`, each aligned with AOC rules and downstream compatibility requirements.
|
||||
- **Surface parity.** Operators can create, monitor, and download exports through the Web API gateway, Console workflows, and the CLI (`stella export ...`). All surfaces enforce tenant scope and RBAC consistently.
|
||||
- **Automation hooks.** One-off, cron, and event triggers are orchestrated via the Scheduler/Orchestrator integration. Export telemetry (durations, bundle size, verification outcomes) feeds structured logs, metrics, and optional OpenTelemetry traces.
|
||||
|
||||
### Profile variants at a glance
|
||||
|
||||
| Profile | Contents | Primary scenarios | Distribution defaults |
|
||||
|---------|----------|-------------------|-----------------------|
|
||||
| `json:raw` | Canonical advisories, VEX, SBOM JSONL with hashes | Downstream analytics, evidence escrow | HTTP download, object storage |
|
||||
| `json:policy` | `json:raw` plus policy snapshot, evaluated findings | Policy attestation, audit packages | HTTP download, object storage |
|
||||
| `trivy:db` / `trivy:java-db` | Trivy-compatible vulnerability databases | Feed external scanners and CI | OCI artifact push, download |
|
||||
| `mirror:full` | Complete evidence, indexes, policy, provenance | Air-gap mirror, disaster recovery | Filesystem bundle, OCI artifact |
|
||||
| `mirror:delta` | Changes relative to prior manifest | Incremental updates to mirrors | Filesystem bundle, OCI artifact |
|
||||
|
||||
## How it works end-to-end
|
||||
1. **Profile & scope resolution.** A profile defines export type, content filters, and bundle settings. Scope selectors target tenants, artifacts, time windows, ecosystems, or SBOM subjects.
|
||||
2. **Ledger collection.** Workers stream canonical data from Findings Ledger, VEX Lens, Conseiller feeds, and SBOM service. Policy exports pin a deterministic policy snapshot from Policy Engine.
|
||||
3. **Adapter execution.** JSON adapters produce normalized `.jsonl.zst` outputs, Trivy adapters translate to the Trivy DB schema, and mirror adapters build filesystem or OCI bundle layouts.
|
||||
4. **Manifesting & provenance.** Every run emits `export.json` (profile, filters, counts, checksums) and `provenance.json` (source artifacts, policy snapshot ids, signature references).
|
||||
5. **Signing & distribution.** Bundles are signed via configured KMS (cosign-compatible) and distributed through HTTP streaming, OCI registry pushes, or object storage staging.
|
||||
|
||||
Refer to `docs/export-center/architecture.md` (Sprint 35 task) for component diagrams and adapter internals once published.
|
||||
|
||||
## Security and compliance guardrails
|
||||
- **AOC alignment.** Exports bundle raw evidence and optional policy evaluations without mutating source content. Policy overlays remain attributed to Policy Engine and are clearly partitioned.
|
||||
- **Tenant isolation.** All queries, manifests, and bundle paths carry tenant identifiers. Cross-tenant exports require explicit signed approval and ship with provenance trails.
|
||||
- **Signing and encryption.** Manifests and payloads are signed using the platform KMS. Mirror profiles support optional in-bundle encryption (age/AES-GCM) with key wrapping.
|
||||
- **Determinism.** Identical inputs yield identical bundles. Timestamps serialize in UTC ISO-8601; manifests include content hashes for audit replay.
|
||||
|
||||
See `docs/security/policy-governance.md` and `docs/ingestion/aggregation-only-contract.md` for broader guardrail context.
|
||||
|
||||
## Operating it offline
|
||||
- **Offline Kit integration.** Air-gapped deployments receive pre-built export profiles and object storage layout templates through the Offline Kit bundles.
|
||||
- **Mirror bundles.** `mirror:full` packages raw evidence, normalized indexes, policy snapshots, and provenance in a portable filesystem layout suitable for disconnected environments. `mirror:delta` tracks changes relative to a prior export manifest.
|
||||
- **No unsanctioned egress.** The exporter respects the platform allowlist. External calls (e.g., OCI pushes) require explicit configuration and are disabled by default for offline installs.
|
||||
|
||||
Consult `docs/24_OFFLINE_KIT.md` for Offline Kit delivery and `docs/ops/concelier-mirror-operations.md` for mirror ingestion procedures.
|
||||
|
||||
## Getting started
|
||||
1. **Choose a profile.** Map requirements to the profile table above. Policy-aware exports need a published policy snapshot.
|
||||
2. **Define selectors.** Decide on tenants, products, SBOM subjects, or time windows to include. Default selectors export the entire tenant scope.
|
||||
3. **Run via preferred surface.**
|
||||
- **Console:** Navigate to the Export Center view, create a run, monitor progress, and download artifacts.
|
||||
- **CLI:** Use `stella export run --profile <name> --selector <filters>` to submit a job, then `stella export download`.
|
||||
- **API:** POST to `/api/export/runs` with profile id and scope payload; stream results from `/api/export/runs/{id}/download`.
|
||||
4. **Verify bundles.** Use the attached provenance manifest and cosign signature to validate contents before distributing downstream.
|
||||
|
||||
Refer to `docs/export-center/cli.md` for detailed command syntax and automation examples.
|
||||
|
||||
## Observability & troubleshooting
|
||||
- Structured logs emit lifecycle events (`fetch`, `adapter`, `sign`, `publish`) with correlation IDs for parallel job tracing.
|
||||
- Metrics `exporter_run_duration_seconds`, `exporter_bundle_bytes_total`, and `exporter_run_failures_total` feed Grafana dashboards defined in the deployment runbooks.
|
||||
- Verification failures or schema mismatches bubble up through failure events and appear in Console/CLI with actionable error messages. Inspect the run's audit log and `provenance.json` for root cause.
|
||||
|
||||
See `docs/observability/policy.md` and `docs/ops/deployment-upgrade-runbook.md` for telemetry and operations guidance.
|
||||
|
||||
> **Imposed rule:** Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.
|
||||
|
||||
@@ -1,139 +1,139 @@
|
||||
# Export Center Profiles
|
||||
|
||||
Export Center profiles define what data is collected, how it is encoded, and which distribution paths are enabled for a run. Profiles are tenant-scoped and deterministic: identical selectors and source data produce identical bundles. This guide summarises built-in profiles, configuration fields, schema conventions, and compatibility notes.
|
||||
|
||||
## Profile catalogue
|
||||
|
||||
| Profile | Kind / Variant | Output artefacts | Primary use cases |
|
||||
|---------|----------------|------------------|-------------------|
|
||||
| `json:raw` | `json` / `raw` | Canonical JSONL archives of advisories, VEX, SBOMs | Evidence escrow, analytics pipelines |
|
||||
| `json:policy` | `json` / `policy` | `json:raw` artefacts plus policy snapshot + evaluated findings | Audit, compliance attestations |
|
||||
| `trivy:db` | `trivy` / `db` | Trivy-compatible vulnerability database | Feeding external scanners / CI |
|
||||
| `trivy:java-db` | `trivy` / `java-db` | Java ecosystem supplement for Trivy | Supply Java CVE data to Trivy |
|
||||
| `mirror:full` | `mirror` / `full` | Complete mirror bundle (raw, policy, indexes, provenance) | Air-gap deployments, disaster recovery |
|
||||
| `mirror:delta` | `mirror` / `delta` | Incremental changes relative to a prior manifest | Efficient mirror updates |
|
||||
|
||||
Profiles can be cloned and customised; configuration is immutable per revision to keep runs reproducible.
|
||||
|
||||
## Common configuration fields
|
||||
|
||||
| Field | Description | Applies to | Notes |
|
||||
|-------|-------------|------------|-------|
|
||||
| `name` | Human-readable identifier displayed in Console/CLI | All | Unique per tenant. |
|
||||
| `kind` | Logical family (`json`, `trivy`, `mirror`) | All | Determines eligible adapters. |
|
||||
| `variant` | Specific export flavour (see table above) | All | Controls adapter behaviour. |
|
||||
| `include` | Record types to include (`advisories`, `vex`, `sboms`, `findings`) | JSON, mirror | Defaults depend on variant. |
|
||||
| `policySnapshotMode` | `required`, `optional`, or `none` | JSON policy, mirror | `required` forces a policy snapshot id when creating runs. |
|
||||
| `distribution` | Enabled distribution drivers (`http`, `oci`, `object`) | All | Offline installs typically disable `oci`. |
|
||||
| `compression` | Compression settings (`zstd`, level) | JSON, mirror | Trivy adapters manage compression internally. |
|
||||
| `encryption` | Mirror encryption options (`enabled`, `recipientKeys`, `strict`) | Mirror | When enabled, only `/data` subtree is encrypted; manifests remain plaintext. |
|
||||
| `retention` | Retention policy (days or `never`) | All | Drives pruning jobs for staged bundles. |
|
||||
|
||||
Selectors (time windows, tenants, products, SBOM subjects, ecosystems) are supplied per run, not stored in the profile.
|
||||
|
||||
## JSON profiles
|
||||
|
||||
### `json:raw`
|
||||
- **Content:** Exports raw advisories, VEX statements, and SBOMs as newline-delimited JSON (`.jsonl.zst`).
|
||||
- **Schema:** Follows canonical StellaOps schema with casing and timestamps normalised. Each record includes `tenant`, `source`, `linkset`, and `content` fields.
|
||||
- **Options:**
|
||||
- `include` defaults to `["advisories", "vex", "sboms"]`.
|
||||
- `compression` defaults to `zstd` level 9.
|
||||
- `maxRecordsPerFile` (optional) splits outputs for large datasets.
|
||||
- **Compatibility:** Intended for analytics platforms, data escrow, or feeding downstream normalisation pipelines.
|
||||
- **Sample manifest excerpt:**
|
||||
|
||||
```json
|
||||
{
|
||||
"profile": { "kind": "json", "variant": "raw" },
|
||||
"outputs": [
|
||||
{ "type": "advisories.jsonl.zst", "sha256": "...", "count": 15234 },
|
||||
{ "type": "vex.jsonl.zst", "sha256": "...", "count": 3045 },
|
||||
{ "type": "sboms.jsonl.zst", "sha256": "...", "count": 872 }
|
||||
],
|
||||
"selectors": { "tenant": "acme", "products": ["registry.example/app"] }
|
||||
}
|
||||
```
|
||||
|
||||
### `json:policy`
|
||||
- **Content:** Everything from `json:raw` plus:
|
||||
- `policy_snapshot.json` (policy metadata, version, hash).
|
||||
- `findings.policy.jsonl.zst` (evaluated findings with decision, rationale, rule id, inputs hash).
|
||||
- **Determinism:** Requires a policy snapshot id; runs fail if snapshot is missing or non-deterministic mode is active.
|
||||
- **Use cases:** Compliance exports, auditor packages, policy attestation archives.
|
||||
- **Guardrails:** AOC boundaries preserved: policy outputs are clearly partitioned from raw evidence.
|
||||
|
||||
## Trivy profiles
|
||||
|
||||
### `trivy:db`
|
||||
- Detailed adapter behaviour is documented in `docs/export-center/trivy-adapter.md`.
|
||||
- **Content:** Produces a Trivy DB-compatible bundle (SQLite database or tarball as required by Trivy version).
|
||||
- **Mapping rules:**
|
||||
- Advisory namespaces mapped to Trivy vendor IDs (e.g., `ubuntu`, `debian`, `npm`).
|
||||
- Version ranges translated into Trivy's semantic version syntax.
|
||||
- Severity mapped to Trivy standard (`UNKNOWN`, `LOW`, `MEDIUM`, `HIGH`, `CRITICAL`).
|
||||
- **Validation:** Adapter enforces supported Trivy schema versions; configuration includes `targetSchemaVersion`.
|
||||
- **Distribution:** Typically pushed to OCI or object storage for downstream scanners; Console download remains available.
|
||||
|
||||
### `trivy:java-db`
|
||||
- Refer to `docs/export-center/trivy-adapter.md` for ecosystem-specific notes.
|
||||
- **Content:** Optional Java ecosystem supplement for Trivy (matching Trivy's separate Java DB).
|
||||
- **Dependencies:** Requires Java advisories in Findings Ledger; run fails with `ERR_EXPORT_EMPTY` if no Java data present and `allowEmpty=false`.
|
||||
- **Compatibility:** Intended for organisations using Trivy's Java plugin or hardened pipelines that split general and Java feeds.
|
||||
|
||||
## Mirror profiles
|
||||
|
||||
### `mirror:full`
|
||||
- Bundle structure and delta strategy are covered in `docs/export-center/mirror-bundles.md`.
|
||||
- **Content:** Complete export with:
|
||||
- Raw advisories, VEX, SBOMs (`/data/raw`).
|
||||
- Policy overlays (`/data/policy`), including evaluated findings and policy snapshots.
|
||||
- Indexes for fast lookup (`/indexes/advisories.pb`, `/indexes/sboms.pb`).
|
||||
- Manifests and provenance (`/manifests/export.json`, `/manifests/provenance.json`).
|
||||
- **Layout:** Deterministic directory structure with hashed filenames to reduce duplication.
|
||||
- **Encryption:** Optional `encryption` block enables age/AES-GCM encryption of `/data`. `strict=true` encrypts everything except `export.json`.
|
||||
- **Use cases:** Air-gap replication, disaster recovery drills, Offline Kit seeding.
|
||||
|
||||
### `mirror:delta`
|
||||
- See `docs/export-center/mirror-bundles.md` for delta mechanics and application order.
|
||||
- **Content:** Includes only changes relative to a base manifest (specified by `baseExportId` when running).
|
||||
- `changed`, `added`, `removed` lists in `manifests/delta.json`.
|
||||
- Incremental indexes capturing only updated subjects.
|
||||
- **Constraints:** Requires the base manifest to exist in object storage or artifact registry accessible to the worker. Fails with `ERR_EXPORT_BASE_MISSING` otherwise.
|
||||
- **Workflow:** Ideal for frequent updates to mirrored environments with limited bandwidth.
|
||||
|
||||
## Compatibility and guardrails
|
||||
- **Aggregation-Only Contract:** All profiles respect AOC boundaries: raw evidence is never mutated. Policy outputs are appended separately with clear provenance.
|
||||
- **Tenant scoping:** Profiles are tenant-specific. Cross-tenant exports require explicit administrative approval and signed justification.
|
||||
- **Retriable runs:** Re-running a profile with identical selectors yields matching manifests and hashes, facilitating verify-on-download workflows.
|
||||
- **Offline operation:** JSON and mirror profiles function in offline mode without additional configuration. Trivy profiles require pre-seeded schema metadata shipped via Offline Kit.
|
||||
- **Quota integration:** Profiles can define run quotas (per tenant per day). Quota exhaustion surfaces as `429 Too Many Requests` with `X-Stella-Quota-*` hints.
|
||||
|
||||
## Example profile definition (CLI)
|
||||
|
||||
```jsonc
|
||||
{
|
||||
"name": "daily-json-raw",
|
||||
"kind": "json",
|
||||
"variant": "raw",
|
||||
"include": ["advisories", "vex", "sboms"],
|
||||
"distribution": ["http", "object"],
|
||||
"compression": { "codec": "zstd", "level": 9 },
|
||||
"retention": { "mode": "days", "value": 14 }
|
||||
}
|
||||
```
|
||||
|
||||
Create via `stella export profile create --file profile.json` (CLI command documented separately).
|
||||
|
||||
## Verification workflow
|
||||
- Download bundle via Console/CLI and extract `export.json` and `provenance.json`.
|
||||
- Run `cosign verify --key <tenant-key> export.json` (for detatched signatures use `--signature export.json.sig`).
|
||||
- Validate Trivy bundles with `trivy --cache-dir <temp> --debug --db-repository <oci-ref>` or local `trivy module db import`.
|
||||
- For mirror bundles, run internal `mirror verify` script (bundled in Offline Kit) to ensure directory layout and digests match manifest.
|
||||
|
||||
## Extending profiles
|
||||
- Use API/CLI to clone an existing profile and adjust `include`, `distribution`, or retention policies.
|
||||
- Adapter plug-ins can introduce new variants (e.g., `json:raw-lite`, `mirror:policy-only`). Plug-ins must be registered at service restart and documented under `/docs/export-center/profiles.md`.
|
||||
- Any new profile must append the imposed rule line and follow determinism and guardrail requirements.
|
||||
|
||||
> **Imposed rule:** Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.
|
||||
# Export Center Profiles
|
||||
|
||||
Export Center profiles define what data is collected, how it is encoded, and which distribution paths are enabled for a run. Profiles are tenant-scoped and deterministic: identical selectors and source data produce identical bundles. This guide summarises built-in profiles, configuration fields, schema conventions, and compatibility notes.
|
||||
|
||||
## Profile catalogue
|
||||
|
||||
| Profile | Kind / Variant | Output artefacts | Primary use cases |
|
||||
|---------|----------------|------------------|-------------------|
|
||||
| `json:raw` | `json` / `raw` | Canonical JSONL archives of advisories, VEX, SBOMs | Evidence escrow, analytics pipelines |
|
||||
| `json:policy` | `json` / `policy` | `json:raw` artefacts plus policy snapshot + evaluated findings | Audit, compliance attestations |
|
||||
| `trivy:db` | `trivy` / `db` | Trivy-compatible vulnerability database | Feeding external scanners / CI |
|
||||
| `trivy:java-db` | `trivy` / `java-db` | Java ecosystem supplement for Trivy | Supply Java CVE data to Trivy |
|
||||
| `mirror:full` | `mirror` / `full` | Complete mirror bundle (raw, policy, indexes, provenance) | Air-gap deployments, disaster recovery |
|
||||
| `mirror:delta` | `mirror` / `delta` | Incremental changes relative to a prior manifest | Efficient mirror updates |
|
||||
|
||||
Profiles can be cloned and customised; configuration is immutable per revision to keep runs reproducible.
|
||||
|
||||
## Common configuration fields
|
||||
|
||||
| Field | Description | Applies to | Notes |
|
||||
|-------|-------------|------------|-------|
|
||||
| `name` | Human-readable identifier displayed in Console/CLI | All | Unique per tenant. |
|
||||
| `kind` | Logical family (`json`, `trivy`, `mirror`) | All | Determines eligible adapters. |
|
||||
| `variant` | Specific export flavour (see table above) | All | Controls adapter behaviour. |
|
||||
| `include` | Record types to include (`advisories`, `vex`, `sboms`, `findings`) | JSON, mirror | Defaults depend on variant. |
|
||||
| `policySnapshotMode` | `required`, `optional`, or `none` | JSON policy, mirror | `required` forces a policy snapshot id when creating runs. |
|
||||
| `distribution` | Enabled distribution drivers (`http`, `oci`, `object`) | All | Offline installs typically disable `oci`. |
|
||||
| `compression` | Compression settings (`zstd`, level) | JSON, mirror | Trivy adapters manage compression internally. |
|
||||
| `encryption` | Mirror encryption options (`enabled`, `recipientKeys`, `strict`) | Mirror | When enabled, only `/data` subtree is encrypted; manifests remain plaintext. |
|
||||
| `retention` | Retention policy (days or `never`) | All | Drives pruning jobs for staged bundles. |
|
||||
|
||||
Selectors (time windows, tenants, products, SBOM subjects, ecosystems) are supplied per run, not stored in the profile.
|
||||
|
||||
## JSON profiles
|
||||
|
||||
### `json:raw`
|
||||
- **Content:** Exports raw advisories, VEX statements, and SBOMs as newline-delimited JSON (`.jsonl.zst`).
|
||||
- **Schema:** Follows canonical StellaOps schema with casing and timestamps normalised. Each record includes `tenant`, `source`, `linkset`, and `content` fields.
|
||||
- **Options:**
|
||||
- `include` defaults to `["advisories", "vex", "sboms"]`.
|
||||
- `compression` defaults to `zstd` level 9.
|
||||
- `maxRecordsPerFile` (optional) splits outputs for large datasets.
|
||||
- **Compatibility:** Intended for analytics platforms, data escrow, or feeding downstream normalisation pipelines.
|
||||
- **Sample manifest excerpt:**
|
||||
|
||||
```json
|
||||
{
|
||||
"profile": { "kind": "json", "variant": "raw" },
|
||||
"outputs": [
|
||||
{ "type": "advisories.jsonl.zst", "sha256": "...", "count": 15234 },
|
||||
{ "type": "vex.jsonl.zst", "sha256": "...", "count": 3045 },
|
||||
{ "type": "sboms.jsonl.zst", "sha256": "...", "count": 872 }
|
||||
],
|
||||
"selectors": { "tenant": "acme", "products": ["registry.example/app"] }
|
||||
}
|
||||
```
|
||||
|
||||
### `json:policy`
|
||||
- **Content:** Everything from `json:raw` plus:
|
||||
- `policy_snapshot.json` (policy metadata, version, hash).
|
||||
- `findings.policy.jsonl.zst` (evaluated findings with decision, rationale, rule id, inputs hash).
|
||||
- **Determinism:** Requires a policy snapshot id; runs fail if snapshot is missing or non-deterministic mode is active.
|
||||
- **Use cases:** Compliance exports, auditor packages, policy attestation archives.
|
||||
- **Guardrails:** AOC boundaries preserved: policy outputs are clearly partitioned from raw evidence.
|
||||
|
||||
## Trivy profiles
|
||||
|
||||
### `trivy:db`
|
||||
- Detailed adapter behaviour is documented in `docs/export-center/trivy-adapter.md`.
|
||||
- **Content:** Produces a Trivy DB-compatible bundle (SQLite database or tarball as required by Trivy version).
|
||||
- **Mapping rules:**
|
||||
- Advisory namespaces mapped to Trivy vendor IDs (e.g., `ubuntu`, `debian`, `npm`).
|
||||
- Version ranges translated into Trivy's semantic version syntax.
|
||||
- Severity mapped to Trivy standard (`UNKNOWN`, `LOW`, `MEDIUM`, `HIGH`, `CRITICAL`).
|
||||
- **Validation:** Adapter enforces supported Trivy schema versions; configuration includes `targetSchemaVersion`.
|
||||
- **Distribution:** Typically pushed to OCI or object storage for downstream scanners; Console download remains available.
|
||||
|
||||
### `trivy:java-db`
|
||||
- Refer to `docs/export-center/trivy-adapter.md` for ecosystem-specific notes.
|
||||
- **Content:** Optional Java ecosystem supplement for Trivy (matching Trivy's separate Java DB).
|
||||
- **Dependencies:** Requires Java advisories in Findings Ledger; run fails with `ERR_EXPORT_EMPTY` if no Java data present and `allowEmpty=false`.
|
||||
- **Compatibility:** Intended for organisations using Trivy's Java plugin or hardened pipelines that split general and Java feeds.
|
||||
|
||||
## Mirror profiles
|
||||
|
||||
### `mirror:full`
|
||||
- Bundle structure and delta strategy are covered in `docs/export-center/mirror-bundles.md`.
|
||||
- **Content:** Complete export with:
|
||||
- Raw advisories, VEX, SBOMs (`/data/raw`).
|
||||
- Policy overlays (`/data/policy`), including evaluated findings and policy snapshots.
|
||||
- Indexes for fast lookup (`/indexes/advisories.pb`, `/indexes/sboms.pb`).
|
||||
- Manifests and provenance (`/manifests/export.json`, `/manifests/provenance.json`).
|
||||
- **Layout:** Deterministic directory structure with hashed filenames to reduce duplication.
|
||||
- **Encryption:** Optional `encryption` block enables age/AES-GCM encryption of `/data`. `strict=true` encrypts everything except `export.json`.
|
||||
- **Use cases:** Air-gap replication, disaster recovery drills, Offline Kit seeding.
|
||||
|
||||
### `mirror:delta`
|
||||
- See `docs/export-center/mirror-bundles.md` for delta mechanics and application order.
|
||||
- **Content:** Includes only changes relative to a base manifest (specified by `baseExportId` when running).
|
||||
- `changed`, `added`, `removed` lists in `manifests/delta.json`.
|
||||
- Incremental indexes capturing only updated subjects.
|
||||
- **Constraints:** Requires the base manifest to exist in object storage or artifact registry accessible to the worker. Fails with `ERR_EXPORT_BASE_MISSING` otherwise.
|
||||
- **Workflow:** Ideal for frequent updates to mirrored environments with limited bandwidth.
|
||||
|
||||
## Compatibility and guardrails
|
||||
- **Aggregation-Only Contract:** All profiles respect AOC boundaries: raw evidence is never mutated. Policy outputs are appended separately with clear provenance.
|
||||
- **Tenant scoping:** Profiles are tenant-specific. Cross-tenant exports require explicit administrative approval and signed justification.
|
||||
- **Retriable runs:** Re-running a profile with identical selectors yields matching manifests and hashes, facilitating verify-on-download workflows.
|
||||
- **Offline operation:** JSON and mirror profiles function in offline mode without additional configuration. Trivy profiles require pre-seeded schema metadata shipped via Offline Kit.
|
||||
- **Quota integration:** Profiles can define run quotas (per tenant per day). Quota exhaustion surfaces as `429 Too Many Requests` with `X-Stella-Quota-*` hints.
|
||||
|
||||
## Example profile definition (CLI)
|
||||
|
||||
```jsonc
|
||||
{
|
||||
"name": "daily-json-raw",
|
||||
"kind": "json",
|
||||
"variant": "raw",
|
||||
"include": ["advisories", "vex", "sboms"],
|
||||
"distribution": ["http", "object"],
|
||||
"compression": { "codec": "zstd", "level": 9 },
|
||||
"retention": { "mode": "days", "value": 14 }
|
||||
}
|
||||
```
|
||||
|
||||
Create via `stella export profile create --file profile.json` (CLI command documented separately).
|
||||
|
||||
## Verification workflow
|
||||
- Download bundle via Console/CLI and extract `export.json` and `provenance.json`.
|
||||
- Run `cosign verify --key <tenant-key> export.json` (for detatched signatures use `--signature export.json.sig`).
|
||||
- Validate Trivy bundles with `trivy --cache-dir <temp> --debug --db-repository <oci-ref>` or local `trivy module db import`.
|
||||
- For mirror bundles, run internal `mirror verify` script (bundled in Offline Kit) to ensure directory layout and digests match manifest.
|
||||
|
||||
## Extending profiles
|
||||
- Use API/CLI to clone an existing profile and adjust `include`, `distribution`, or retention policies.
|
||||
- Adapter plug-ins can introduce new variants (e.g., `json:raw-lite`, `mirror:policy-only`). Plug-ins must be registered at service restart and documented under `/docs/export-center/profiles.md`.
|
||||
- Any new profile must append the imposed rule line and follow determinism and guardrail requirements.
|
||||
|
||||
> **Imposed rule:** Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.
|
||||
|
||||
@@ -1,150 +1,150 @@
|
||||
# Export Center Provenance & Signing
|
||||
|
||||
> **Imposed rule:** Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.
|
||||
|
||||
Export Center runs emit deterministic manifests, provenance records, and signatures so operators can prove bundle integrity end-to-end—whether the artefact is downloaded over HTTPS, pulled as an OCI object, or staged through the Offline Kit. This guide captures the canonical artefacts, signing pipeline, verification workflows, and failure handling expectations that backlogs `EXPORT-SVC-35-005` and `EXPORT-SVC-37-002` implement.
|
||||
|
||||
---
|
||||
|
||||
## 1. Goals & scope
|
||||
|
||||
- **Authenticity.** Every export manifest and provenance document is signed using Authority-managed KMS keys (cosign-compatible) with optional SLSA Level 3 attestation.
|
||||
- **Traceability.** Provenance links each bundle to the inputs that produced it: tenant, findings ledger queries, policy snapshots, SBOM identifiers, adapter versions, and encryption recipients.
|
||||
- **Determinism.** Canonical JSON (sorted keys, RFC 3339 UTC timestamps, normalized numbers) guarantees byte-for-byte stability across reruns with identical input.
|
||||
- **Portability.** Signatures and attestations travel with filesystem bundles, OCI artefacts, and Offline Kit staging trees. Verification does not require online Authority access when the bundle includes the cosign public key.
|
||||
|
||||
---
|
||||
|
||||
## 2. Artefact inventory
|
||||
|
||||
| File | Location | Description | Notes |
|
||||
|------|----------|-------------|-------|
|
||||
| `export.json` | `manifests/export.json` or HTTP `GET /api/export/runs/{id}/manifest` | Canonical manifest describing profile, selectors, counts, SHA-256 digests, compression hints, distribution targets. | Hash of this file is included in provenance `subjects[]`. |
|
||||
| `provenance.json` | `manifests/provenance.json` or `GET /api/export/runs/{id}/provenance` | In-toto provenance record listing subjects, materials, toolchain metadata, encryption recipients, and KMS key identifiers. | Mirrors SLSA Level 2 schema; optionally upgraded to Level 3 with builder attestations. |
|
||||
| `export.json.sig` / `export.json.dsse` | `signatures/export.json.sig` | Cosign signature (and optional DSSE envelope) for manifest. | File naming matches cosign defaults; offline verification scripts expect `.sig`. |
|
||||
| `provenance.json.sig` / `provenance.json.dsse` | `signatures/provenance.json.sig` | Cosign signature (and optional DSSE envelope) for provenance document. | `dsse` present when SLSA Level 3 is enabled. |
|
||||
| `bundle.attestation` | `signatures/bundle.attestation` (optional) | SLSA Level 2/3 attestation binding bundle tarball/OCI digest to the run. | Only produced when `export.attestation.enabled=true`. |
|
||||
| `manifest.yaml` | bundle root | Human-readable summary including digests, sizes, encryption metadata, and verification hints. | Unsigned but redundant; signatures cover the JSON manifests. |
|
||||
|
||||
All digests use lowercase hex SHA-256 (`sha256:<digest>`). When bundle encryption is enabled, `provenance.json` records wrapped data keys and recipient fingerprints under `encryption.recipients[]`.
|
||||
|
||||
---
|
||||
|
||||
## 3. Signing pipeline
|
||||
|
||||
1. **Canonicalisation.** Export worker serialises `export.json` and `provenance.json` using `NotifyCanonicalJsonSerializer` (identical canonical JSON helpers shared across services). Keys are sorted lexicographically, arrays ordered deterministically, timestamps normalised to UTC.
|
||||
2. **Digest creation.** SHA-256 digests are computed and recorded:
|
||||
- `manifest_hash` and `provenance_hash` stored in the run metadata (Mongo) and exported via `/api/export/runs/{id}`.
|
||||
- Provenance `subjects[]` contains both manifest hash and bundle/archive hash.
|
||||
3. **Key retrieval.** Worker obtains a short-lived signing token from Authority’s KMS client using tenant-scoped credentials (`export.sign` scope). Keys live in Authority or tenant-specific HSMs depending on deployment.
|
||||
4. **Signature emission.** Cosign generates detached signatures (`*.sig`). If DSSE is enabled, cosign wraps payload bytes in a DSSE envelope (`*.dsse`). Attestations follow the SLSA Level 2 provenance template; Level 3 requires builder metadata (`EXPORT-SVC-37-002` optional feature flag).
|
||||
5. **Storage & distribution.** Signatures and attestations are written alongside manifests in object storage, included in filesystem bundles, and attached as OCI artefact layers/annotations.
|
||||
6. **Audit trail.** Run metadata captures signer identity (`signing_key_id`), cosign certificate serial, signature timestamps, and verification hints. Console/CLI surface these details for downstream automation.
|
||||
|
||||
> **Key management.** Secrets and key references are configured per tenant via `export.signing`, pointing to Authority clients or external HSM aliases. Offline deployments pre-load cosign public keys into the bundle (`signatures/pubkeys/{tenant}.pem`).
|
||||
|
||||
---
|
||||
|
||||
## 4. Provenance schema highlights
|
||||
|
||||
`provenance.json` follows the SLSA provenance (`https://slsa.dev/provenance/v1`) structure with StellaOps-specific extensions. Key fields:
|
||||
|
||||
| Path | Description |
|
||||
|------|-------------|
|
||||
| `subject[]` | Array of `{name,digest}` pairs. Includes bundle tarball/OCI digest and `export.json` digest. |
|
||||
| `predicateType` | SLSA v1 (default). |
|
||||
| `predicate.builder` | `{id:"stellaops/export-center@<region>"}` identifies the worker instance/cluster. |
|
||||
| `predicate.buildType` | Profile identifier (`mirror:full`, `mirror:delta`, etc.). |
|
||||
| `predicate.invocation.parameters` | Profile selectors, retention flags, encryption mode, base export references. |
|
||||
| `predicate.materials[]` | Source artefacts with digests: findings ledger query snapshots, policy snapshot IDs + hashes, SBOM identifiers, adapter release digests. |
|
||||
| `predicate.metadata.buildFinishedOn` | RFC 3339 timestamp when signing completed. |
|
||||
| `predicate.metadata.reproducible` | Always `true`—workers guarantee determinism. |
|
||||
| `predicate.environment.encryption` | Records encryption recipients, wrapped keys, algorithm (`age` or `aes-gcm`). |
|
||||
| `predicate.environment.kms` | Signing key identifier (`authority://tenant/export-signing-key`) and certificate chain fingerprints. |
|
||||
|
||||
Sample (abridged):
|
||||
|
||||
```json
|
||||
{
|
||||
"subject": [
|
||||
{ "name": "bundle.tar.zst", "digest": { "sha256": "c1fe..." } },
|
||||
{ "name": "manifests/export.json", "digest": { "sha256": "ad42..." } }
|
||||
],
|
||||
"predicate": {
|
||||
"buildType": "mirror:delta",
|
||||
"invocation": {
|
||||
"parameters": {
|
||||
"tenant": "tenant-01",
|
||||
"baseExportId": "run-20251020-01",
|
||||
"selectors": { "sources": ["concelier","vexer"], "profiles": ["mirror"] }
|
||||
}
|
||||
},
|
||||
"materials": [
|
||||
{ "uri": "ledger://tenant-01/findings?cursor=rev-42", "digest": { "sha256": "0f9a..." } },
|
||||
{ "uri": "policy://tenant-01/snapshots/rev-17", "digest": { "sha256": "8c3d..." } }
|
||||
],
|
||||
"environment": {
|
||||
"encryption": {
|
||||
"mode": "age",
|
||||
"recipients": [
|
||||
{ "recipient": "age1qxyz...", "wrappedKey": "BASE64...", "keyId": "tenant-01/notify-age" }
|
||||
]
|
||||
},
|
||||
"kms": {
|
||||
"signingKeyId": "authority://tenant-01/export-signing",
|
||||
"certificateChainSha256": "1f5e..."
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 5. Verification workflows
|
||||
|
||||
| Scenario | Steps |
|
||||
|----------|-------|
|
||||
| **CLI verification** | 1. `stella export manifest <runId> --output manifests/export.json --signature manifests/export.json.sig`<br>2. `stella export provenance <runId> --output manifests/provenance.json --signature manifests/provenance.json.sig`<br>3. `cosign verify-blob --key pubkeys/tenant.pem --signature manifests/export.json.sig manifests/export.json`<br>4. `cosign verify-blob --key pubkeys/tenant.pem --signature manifests/provenance.json.sig manifests/provenance.json` |
|
||||
| **Bundle verification (offline)** | 1. Extract bundle (or mount OCI artefact).<br>2. Validate manifest/provenance signatures using bundled public key.<br>3. Recompute SHA-256 for `data/` files and compare with entries in `export.json`.<br>4. If encrypted, decrypt with Age/AES-GCM recipient key, then re-run digest comparisons on decrypted content. |
|
||||
| **CI pipeline** | Use `stella export verify --manifest manifests/export.json --provenance manifests/provenance.json --signature manifests/export.json.sig --signature manifests/provenance.json.sig` (task `CLI-EXPORT-37-001`). Failure exits non-zero with reason codes (`ERR_EXPORT_SIG_INVALID`, `ERR_EXPORT_DIGEST_MISMATCH`). |
|
||||
| **Console download** | Console automatically verifies signatures before exposing the bundle; failure surfaces an actionable error referencing the export run ID and required remediation. |
|
||||
|
||||
Verification guidance (docs/cli/cli-reference.md §export) cross-links here; keep both docs in sync when CLI behaviour changes.
|
||||
|
||||
---
|
||||
|
||||
## 6. Distribution considerations
|
||||
|
||||
- **HTTP headers.** `X-Export-Digest` includes bundle digest; `X-Export-Provenance` references `provenance.json` URL; `X-Export-Signature` references `.sig`. Clients use these hints to short-circuit re-downloads.
|
||||
- **OCI annotations.** `org.opencontainers.image.ref.name`, `io.stellaops.export.manifest-digest`, and `io.stellaops.export.provenance-ref` allow registry tooling to locate manifests/signatures quickly.
|
||||
- **Offline Kit staging.** Offline kit assembler copies `manifests/`, `signatures/`, and `pubkeys/` verbatim. Verification scripts (`offline-kits/bin/verify-export.sh`) wrap the cosign commands described above.
|
||||
|
||||
---
|
||||
|
||||
## 7. Failure handling & observability
|
||||
|
||||
- Runs surface signature status via `/api/export/runs/{id}` (`signing.status`, `signing.lastError`). Common errors include `ERR_EXPORT_KMS_UNAVAILABLE`, `ERR_EXPORT_ATTESTATION_FAILED`, `ERR_EXPORT_CANONICALIZE`.
|
||||
- Metrics: `exporter_sign_duration_seconds`, `exporter_sign_failures_total{error_code}`, `exporter_provenance_verify_failures_total`.
|
||||
- Logs: `phase=sign`, `error_code`, `signing_key_id`, `cosign_certificate_sn`.
|
||||
- Alerts: DevOps dashboards (task `DEVOPS-EXPORT-37-001`) trigger on consecutive signing failures or verification failures >0.
|
||||
|
||||
When verification fails downstream, operators should:
|
||||
1. Confirm signatures using the known-good key.
|
||||
2. Inspect `provenance.json` materials; rerun the source queries to ensure matching digests.
|
||||
3. Review run audit logs and retry export with `--resume` to regenerate manifests.
|
||||
|
||||
---
|
||||
|
||||
## 8. Compliance checklist
|
||||
|
||||
- [ ] Manifests and provenance documents generated with canonical JSON, deterministic digests, and signatures.
|
||||
- [ ] Cosign public keys published per tenant, rotated through Authority, and distributed to Offline Kit consumers.
|
||||
- [ ] SLSA attestations enabled where supply-chain requirements demand Level 3 evidence.
|
||||
- [ ] CLI/Console verification paths documented and tested (CI pipelines exercise `stella export verify`).
|
||||
- [ ] Encryption metadata (recipients, wrapped keys) recorded in provenance and validated during verification.
|
||||
- [ ] Run audit logs capture signature timestamps, signer identity, and failure reasons.
|
||||
|
||||
---
|
||||
|
||||
> **Imposed rule reminder:** Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.
|
||||
# Export Center Provenance & Signing
|
||||
|
||||
> **Imposed rule:** Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.
|
||||
|
||||
Export Center runs emit deterministic manifests, provenance records, and signatures so operators can prove bundle integrity end-to-end—whether the artefact is downloaded over HTTPS, pulled as an OCI object, or staged through the Offline Kit. This guide captures the canonical artefacts, signing pipeline, verification workflows, and failure handling expectations that backlogs `EXPORT-SVC-35-005` and `EXPORT-SVC-37-002` implement.
|
||||
|
||||
---
|
||||
|
||||
## 1. Goals & scope
|
||||
|
||||
- **Authenticity.** Every export manifest and provenance document is signed using Authority-managed KMS keys (cosign-compatible) with optional SLSA Level 3 attestation.
|
||||
- **Traceability.** Provenance links each bundle to the inputs that produced it: tenant, findings ledger queries, policy snapshots, SBOM identifiers, adapter versions, and encryption recipients.
|
||||
- **Determinism.** Canonical JSON (sorted keys, RFC 3339 UTC timestamps, normalized numbers) guarantees byte-for-byte stability across reruns with identical input.
|
||||
- **Portability.** Signatures and attestations travel with filesystem bundles, OCI artefacts, and Offline Kit staging trees. Verification does not require online Authority access when the bundle includes the cosign public key.
|
||||
|
||||
---
|
||||
|
||||
## 2. Artefact inventory
|
||||
|
||||
| File | Location | Description | Notes |
|
||||
|------|----------|-------------|-------|
|
||||
| `export.json` | `manifests/export.json` or HTTP `GET /api/export/runs/{id}/manifest` | Canonical manifest describing profile, selectors, counts, SHA-256 digests, compression hints, distribution targets. | Hash of this file is included in provenance `subjects[]`. |
|
||||
| `provenance.json` | `manifests/provenance.json` or `GET /api/export/runs/{id}/provenance` | In-toto provenance record listing subjects, materials, toolchain metadata, encryption recipients, and KMS key identifiers. | Mirrors SLSA Level 2 schema; optionally upgraded to Level 3 with builder attestations. |
|
||||
| `export.json.sig` / `export.json.dsse` | `signatures/export.json.sig` | Cosign signature (and optional DSSE envelope) for manifest. | File naming matches cosign defaults; offline verification scripts expect `.sig`. |
|
||||
| `provenance.json.sig` / `provenance.json.dsse` | `signatures/provenance.json.sig` | Cosign signature (and optional DSSE envelope) for provenance document. | `dsse` present when SLSA Level 3 is enabled. |
|
||||
| `bundle.attestation` | `signatures/bundle.attestation` (optional) | SLSA Level 2/3 attestation binding bundle tarball/OCI digest to the run. | Only produced when `export.attestation.enabled=true`. |
|
||||
| `manifest.yaml` | bundle root | Human-readable summary including digests, sizes, encryption metadata, and verification hints. | Unsigned but redundant; signatures cover the JSON manifests. |
|
||||
|
||||
All digests use lowercase hex SHA-256 (`sha256:<digest>`). When bundle encryption is enabled, `provenance.json` records wrapped data keys and recipient fingerprints under `encryption.recipients[]`.
|
||||
|
||||
---
|
||||
|
||||
## 3. Signing pipeline
|
||||
|
||||
1. **Canonicalisation.** Export worker serialises `export.json` and `provenance.json` using `NotifyCanonicalJsonSerializer` (identical canonical JSON helpers shared across services). Keys are sorted lexicographically, arrays ordered deterministically, timestamps normalised to UTC.
|
||||
2. **Digest creation.** SHA-256 digests are computed and recorded:
|
||||
- `manifest_hash` and `provenance_hash` stored in the run metadata (Mongo) and exported via `/api/export/runs/{id}`.
|
||||
- Provenance `subjects[]` contains both manifest hash and bundle/archive hash.
|
||||
3. **Key retrieval.** Worker obtains a short-lived signing token from Authority’s KMS client using tenant-scoped credentials (`export.sign` scope). Keys live in Authority or tenant-specific HSMs depending on deployment.
|
||||
4. **Signature emission.** Cosign generates detached signatures (`*.sig`). If DSSE is enabled, cosign wraps payload bytes in a DSSE envelope (`*.dsse`). Attestations follow the SLSA Level 2 provenance template; Level 3 requires builder metadata (`EXPORT-SVC-37-002` optional feature flag).
|
||||
5. **Storage & distribution.** Signatures and attestations are written alongside manifests in object storage, included in filesystem bundles, and attached as OCI artefact layers/annotations.
|
||||
6. **Audit trail.** Run metadata captures signer identity (`signing_key_id`), cosign certificate serial, signature timestamps, and verification hints. Console/CLI surface these details for downstream automation.
|
||||
|
||||
> **Key management.** Secrets and key references are configured per tenant via `export.signing`, pointing to Authority clients or external HSM aliases. Offline deployments pre-load cosign public keys into the bundle (`signatures/pubkeys/{tenant}.pem`).
|
||||
|
||||
---
|
||||
|
||||
## 4. Provenance schema highlights
|
||||
|
||||
`provenance.json` follows the SLSA provenance (`https://slsa.dev/provenance/v1`) structure with StellaOps-specific extensions. Key fields:
|
||||
|
||||
| Path | Description |
|
||||
|------|-------------|
|
||||
| `subject[]` | Array of `{name,digest}` pairs. Includes bundle tarball/OCI digest and `export.json` digest. |
|
||||
| `predicateType` | SLSA v1 (default). |
|
||||
| `predicate.builder` | `{id:"stellaops/export-center@<region>"}` identifies the worker instance/cluster. |
|
||||
| `predicate.buildType` | Profile identifier (`mirror:full`, `mirror:delta`, etc.). |
|
||||
| `predicate.invocation.parameters` | Profile selectors, retention flags, encryption mode, base export references. |
|
||||
| `predicate.materials[]` | Source artefacts with digests: findings ledger query snapshots, policy snapshot IDs + hashes, SBOM identifiers, adapter release digests. |
|
||||
| `predicate.metadata.buildFinishedOn` | RFC 3339 timestamp when signing completed. |
|
||||
| `predicate.metadata.reproducible` | Always `true`—workers guarantee determinism. |
|
||||
| `predicate.environment.encryption` | Records encryption recipients, wrapped keys, algorithm (`age` or `aes-gcm`). |
|
||||
| `predicate.environment.kms` | Signing key identifier (`authority://tenant/export-signing-key`) and certificate chain fingerprints. |
|
||||
|
||||
Sample (abridged):
|
||||
|
||||
```json
|
||||
{
|
||||
"subject": [
|
||||
{ "name": "bundle.tar.zst", "digest": { "sha256": "c1fe..." } },
|
||||
{ "name": "manifests/export.json", "digest": { "sha256": "ad42..." } }
|
||||
],
|
||||
"predicate": {
|
||||
"buildType": "mirror:delta",
|
||||
"invocation": {
|
||||
"parameters": {
|
||||
"tenant": "tenant-01",
|
||||
"baseExportId": "run-20251020-01",
|
||||
"selectors": { "sources": ["concelier","vexer"], "profiles": ["mirror"] }
|
||||
}
|
||||
},
|
||||
"materials": [
|
||||
{ "uri": "ledger://tenant-01/findings?cursor=rev-42", "digest": { "sha256": "0f9a..." } },
|
||||
{ "uri": "policy://tenant-01/snapshots/rev-17", "digest": { "sha256": "8c3d..." } }
|
||||
],
|
||||
"environment": {
|
||||
"encryption": {
|
||||
"mode": "age",
|
||||
"recipients": [
|
||||
{ "recipient": "age1qxyz...", "wrappedKey": "BASE64...", "keyId": "tenant-01/notify-age" }
|
||||
]
|
||||
},
|
||||
"kms": {
|
||||
"signingKeyId": "authority://tenant-01/export-signing",
|
||||
"certificateChainSha256": "1f5e..."
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 5. Verification workflows
|
||||
|
||||
| Scenario | Steps |
|
||||
|----------|-------|
|
||||
| **CLI verification** | 1. `stella export manifest <runId> --output manifests/export.json --signature manifests/export.json.sig`<br>2. `stella export provenance <runId> --output manifests/provenance.json --signature manifests/provenance.json.sig`<br>3. `cosign verify-blob --key pubkeys/tenant.pem --signature manifests/export.json.sig manifests/export.json`<br>4. `cosign verify-blob --key pubkeys/tenant.pem --signature manifests/provenance.json.sig manifests/provenance.json` |
|
||||
| **Bundle verification (offline)** | 1. Extract bundle (or mount OCI artefact).<br>2. Validate manifest/provenance signatures using bundled public key.<br>3. Recompute SHA-256 for `data/` files and compare with entries in `export.json`.<br>4. If encrypted, decrypt with Age/AES-GCM recipient key, then re-run digest comparisons on decrypted content. |
|
||||
| **CI pipeline** | Use `stella export verify --manifest manifests/export.json --provenance manifests/provenance.json --signature manifests/export.json.sig --signature manifests/provenance.json.sig` (task `CLI-EXPORT-37-001`). Failure exits non-zero with reason codes (`ERR_EXPORT_SIG_INVALID`, `ERR_EXPORT_DIGEST_MISMATCH`). |
|
||||
| **Console download** | Console automatically verifies signatures before exposing the bundle; failure surfaces an actionable error referencing the export run ID and required remediation. |
|
||||
|
||||
Verification guidance (docs/cli/cli-reference.md §export) cross-links here; keep both docs in sync when CLI behaviour changes.
|
||||
|
||||
---
|
||||
|
||||
## 6. Distribution considerations
|
||||
|
||||
- **HTTP headers.** `X-Export-Digest` includes bundle digest; `X-Export-Provenance` references `provenance.json` URL; `X-Export-Signature` references `.sig`. Clients use these hints to short-circuit re-downloads.
|
||||
- **OCI annotations.** `org.opencontainers.image.ref.name`, `io.stellaops.export.manifest-digest`, and `io.stellaops.export.provenance-ref` allow registry tooling to locate manifests/signatures quickly.
|
||||
- **Offline Kit staging.** Offline kit assembler copies `manifests/`, `signatures/`, and `pubkeys/` verbatim. Verification scripts (`offline-kits/bin/verify-export.sh`) wrap the cosign commands described above.
|
||||
|
||||
---
|
||||
|
||||
## 7. Failure handling & observability
|
||||
|
||||
- Runs surface signature status via `/api/export/runs/{id}` (`signing.status`, `signing.lastError`). Common errors include `ERR_EXPORT_KMS_UNAVAILABLE`, `ERR_EXPORT_ATTESTATION_FAILED`, `ERR_EXPORT_CANONICALIZE`.
|
||||
- Metrics: `exporter_sign_duration_seconds`, `exporter_sign_failures_total{error_code}`, `exporter_provenance_verify_failures_total`.
|
||||
- Logs: `phase=sign`, `error_code`, `signing_key_id`, `cosign_certificate_sn`.
|
||||
- Alerts: DevOps dashboards (task `DEVOPS-EXPORT-37-001`) trigger on consecutive signing failures or verification failures >0.
|
||||
|
||||
When verification fails downstream, operators should:
|
||||
1. Confirm signatures using the known-good key.
|
||||
2. Inspect `provenance.json` materials; rerun the source queries to ensure matching digests.
|
||||
3. Review run audit logs and retry export with `--resume` to regenerate manifests.
|
||||
|
||||
---
|
||||
|
||||
## 8. Compliance checklist
|
||||
|
||||
- [ ] Manifests and provenance documents generated with canonical JSON, deterministic digests, and signatures.
|
||||
- [ ] Cosign public keys published per tenant, rotated through Authority, and distributed to Offline Kit consumers.
|
||||
- [ ] SLSA attestations enabled where supply-chain requirements demand Level 3 evidence.
|
||||
- [ ] CLI/Console verification paths documented and tested (CI pipelines exercise `stella export verify`).
|
||||
- [ ] Encryption metadata (recipients, wrapped keys) recorded in provenance and validated during verification.
|
||||
- [ ] Run audit logs capture signature timestamps, signer identity, and failure reasons.
|
||||
|
||||
---
|
||||
|
||||
> **Imposed rule reminder:** Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.
|
||||
|
||||
@@ -1,246 +1,246 @@
|
||||
# Export Center Trivy Adapters
|
||||
|
||||
The Trivy adapters translate StellaOps normalized advisories into the format consumed by Aqua Security's Trivy scanner. They enable downstream tooling to reuse StellaOps' curated data without bespoke converters, while preserving Aggregation-Only Contract (AOC) boundaries. This guide documents bundle layouts, field mappings, compatibility guarantees, validation workflows, and configuration toggles introduced in Sprint 36 (`EXPORT-SVC-36-001`, `EXPORT-SVC-36-002`).
|
||||
|
||||
> The current Export Center build is wiring the API and workers. Treat this document as the canonical interface for adapter implementation and update any behavioural changes during task sign-off.
|
||||
|
||||
## 1. Adapter overview
|
||||
|
||||
| Variant | Bundle | Default profile | Notes |
|
||||
|---------|--------|-----------------|-------|
|
||||
| `trivy:db` | `db.bundle` | `trivy:db` | Core vulnerability database compatible with Trivy CLI >= 0.50.0 (schema v2). |
|
||||
| `trivy:java-db` | `java-db.bundle` | Optional extension | Java ecosystem supplement (Maven, Gradle). Enabled when `ExportCenter:Profiles:Trivy:EnableJavaDb=true`. |
|
||||
|
||||
Both variants ship inside the export run under `/export/trivy/`. Each bundle is a gzip-compressed tarball containing:
|
||||
|
||||
```
|
||||
metadata.json
|
||||
trivy.db # BoltDB file with vulnerability/provider tables
|
||||
packages/*.json # Only when schema requires JSON overlays (language ecosystems)
|
||||
```
|
||||
|
||||
The adapters never mutate input evidence. They only reshape normalized advisories and copy the exact upstream references so consumers can trace provenance.
|
||||
|
||||
## 2. Bundle layout
|
||||
|
||||
```
|
||||
trivy/
|
||||
db.bundle
|
||||
+-- metadata.json
|
||||
+-- trivy.db
|
||||
java-db.bundle # present when Java DB enabled
|
||||
+-- metadata.json
|
||||
+-- trivy-java.db
|
||||
+-- ecosystem/...
|
||||
signatures/
|
||||
trivy-db.sig
|
||||
trivy-java-db.sig
|
||||
```
|
||||
|
||||
`metadata.json` aligns with Trivy's expectations (`schemaVersion`, `buildInfo`, `updatedAt`, etc.). Export Center adds an `stella` block to capture profile id, run id, and policy snapshot hints.
|
||||
|
||||
Example `metadata.json` (trimmed):
|
||||
|
||||
```json
|
||||
{
|
||||
"schemaVersion": 2,
|
||||
"buildInfo": {
|
||||
"trivyVersion": "0.50.1",
|
||||
"vulnerabilityDBVersion": "2025-10-28T00:00:00Z"
|
||||
},
|
||||
"updatedAt": "2025-10-29T11:42:03Z",
|
||||
"stella": {
|
||||
"runId": "run-20251029-01",
|
||||
"profileId": "prof-trivy-db",
|
||||
"tenant": "acme",
|
||||
"policySnapshotId": "policy-snap-42",
|
||||
"schemaVersion": 2
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## 3. Field mappings
|
||||
|
||||
### 3.1 Namespace resolution
|
||||
|
||||
| Stella field | Trivy field | Notes |
|
||||
|--------------|-------------|-------|
|
||||
| `advisory.source.vendor` | `namespace` | Canonicalized to lowercase; e.g. `Ubuntu` -> `ubuntu`. |
|
||||
| `advisory.source.product` | `distribution` / `ecosystem` | Mapped via allowlist (`Ubuntu 22.04` -> `ubuntu:22.04`). |
|
||||
| `package.ecosystem` | `package.ecosystem` | OSS ecosystems (`npm`, `pip`, `nuget`, etc.). |
|
||||
| `package.nevra` / `package.evr` | `package.version` (OS) | RPM/DEB version semantics preserved. |
|
||||
|
||||
If a record lacks a supported namespace, the adapter drops it and logs `adapter.trivy.unsupported_namespace`.
|
||||
|
||||
### 3.2 Vulnerability metadata
|
||||
|
||||
| Stella field | Trivy field | Transformation |
|
||||
|--------------|-------------|----------------|
|
||||
| `advisory.identifiers.cve[]` | `vulnerability.CVEIDs` | Array of strings. |
|
||||
| `advisory.identifiers.aliases[]` | `vulnerability.CWEIDs` / `References` | CVE -> `CVEIDs`, others appended to `References`. |
|
||||
| `advisory.summary` | `vulnerability.Title` | Stripped to 256 chars; rest moved to `Description`. |
|
||||
| `advisory.description` | `vulnerability.Description` | Markdown allowed, normalized to LF line endings. |
|
||||
| `advisory.severity.normalized` | `vulnerability.Severity` | Uses table below. |
|
||||
| `advisory.cvss[]` | `vulnerability.CVSS` | Stored as `{"vector": "...", "score": 7.8, "source": "NVD"}`. |
|
||||
| `advisory.published` | `vulnerability.PublishedDate` | ISO 8601 UTC. |
|
||||
| `advisory.modified` | `vulnerability.LastModifiedDate` | ISO 8601 UTC. |
|
||||
| `advisory.vendorStatement` | `vulnerability.VendorSeverity` / `VendorVectors` | Preserved in vendor block. |
|
||||
|
||||
Severity mapping:
|
||||
|
||||
| Stella severity | Trivy severity |
|
||||
|-----------------|----------------|
|
||||
| `critical` | `CRITICAL` |
|
||||
| `high` | `HIGH` |
|
||||
| `medium` | `MEDIUM` |
|
||||
| `low` | `LOW` |
|
||||
| `none` / `info` | `UNKNOWN` |
|
||||
|
||||
### 3.3 Affected packages
|
||||
|
||||
| Stella field | Trivy field | Notes |
|
||||
|--------------|-------------|-------|
|
||||
| `package.name` | `package.name` | For OS distros uses source package when available. |
|
||||
| `package.purl` | `package.PURL` | Copied verbatim. |
|
||||
| `affects.vulnerableRange` | `package.vulnerableVersionRange` | SemVer or distro version range. |
|
||||
| `remediations.fixedVersion` | `package.fixedVersion` | Latest known fix. |
|
||||
| `remediations.urls[]` | `package.links` | Array; duplicates removed. |
|
||||
| `states.cpes[]` | `package.cpes` | For CPE-backed advisories. |
|
||||
|
||||
The adapter deduplicates entries by `(namespace, package.name, vulnerableRange)` to avoid duplicate records when multiple upstream segments agree.
|
||||
|
||||
Example mapping (Ubuntu advisory):
|
||||
|
||||
```jsonc
|
||||
// Stella normalized input
|
||||
{
|
||||
"source": {"vendor": "Ubuntu", "product": "22.04"},
|
||||
"identifiers": {"cve": ["CVE-2024-12345"]},
|
||||
"severity": {"normalized": "high"},
|
||||
"affects": [{
|
||||
"package": {"name": "openssl", "ecosystem": "ubuntu", "nevra": "1.1.1f-1ubuntu2.12"},
|
||||
"vulnerableRange": "< 1.1.1f-1ubuntu2.13",
|
||||
"remediations": [{"fixedVersion": "1.1.1f-1ubuntu2.13"}]
|
||||
}]
|
||||
}
|
||||
|
||||
// Trivy vulnerability entry
|
||||
{
|
||||
"namespace": "ubuntu",
|
||||
"package": {
|
||||
"name": "openssl",
|
||||
"version": "< 1.1.1f-1ubuntu2.13",
|
||||
"fixedVersion": "1.1.1f-1ubuntu2.13"
|
||||
},
|
||||
"vulnerability": {
|
||||
"ID": "CVE-2024-12345",
|
||||
"Severity": "HIGH"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 3.4 Java DB specifics
|
||||
|
||||
The Java supplement only includes ecosystems `maven`, `gradle`, `sbt`. Additional fields:
|
||||
|
||||
| Stella field | Trivy Java field | Notes |
|
||||
|--------------|------------------|-------|
|
||||
| `package.group` | `GroupID` | Derived from Maven coordinates. |
|
||||
| `package.artifact` | `ArtifactID` | Derived from Maven coordinates. |
|
||||
| `package.version` | `Version` | Compared with semver-lite rules. |
|
||||
| `affects.symbolicRanges[]` | `VulnerableVersions` | Strings like `[1.0.0,1.2.3)`. |
|
||||
|
||||
## 4. Compatibility matrix
|
||||
|
||||
| Trivy version | Schema version | Supported by adapter | Notes |
|
||||
|---------------|----------------|----------------------|-------|
|
||||
| 0.46.x | 2 | Yes | Baseline compatibility target. |
|
||||
| 0.50.x | 2 | Yes | Default validation target in CI. |
|
||||
| 0.51.x+ | 3 | Pending | Adapter throws `ERR_EXPORT_UNSUPPORTED_SCHEMA` until implemented. |
|
||||
|
||||
Schema mismatches emit `adapter.trivy.unsupported_schema_version` and abort the run. Operators can pin the schema via `ExportCenter:Adapters:Trivy:SchemaVersion`.
|
||||
|
||||
## 5. Validation workflow
|
||||
|
||||
1. **Unit tests** (`StellaOps.ExportCenter.Tests`):
|
||||
- Mapping tests for OS and ecosystem packages.
|
||||
- Severity conversion and range handling property tests.
|
||||
2. **Integration tests** (`EXPORT-SVC-36-001`):
|
||||
- Generate bundle from fixture dataset.
|
||||
- Run `trivy module db import <bundle>` (Trivy CLI) to ensure the bundle is accepted.
|
||||
- For Java DB, run `trivy java-repo --db <bundle>` against sample repository.
|
||||
3. **CI smoke (`DEVOPS-EXPORT-36-001`)**:
|
||||
- Validate metadata fields using `jq`.
|
||||
- Ensure signatures verify with `cosign`.
|
||||
- Check runtime by invoking `trivy fs --cache-dir <temp> --skip-update --custom-db <bundle> fixtures/image`.
|
||||
|
||||
Failures set the run status to `failed` with `errorCode="adapter-trivy"` so Console/CLI expose the reason.
|
||||
|
||||
## 6. Configuration knobs
|
||||
|
||||
```yaml
|
||||
ExportCenter:
|
||||
Adapters:
|
||||
Trivy:
|
||||
SchemaVersion: 2 # enforce schema version
|
||||
IncludeJavaDb: true # enable java-db.bundle
|
||||
AllowEmpty: false # fail when no records match
|
||||
MaxCvssVectorsPerEntry: 5 # truncate to avoid oversized payloads
|
||||
Distribution:
|
||||
Oras:
|
||||
TrivyRepository: "registry.example.com/stella/trivy-db"
|
||||
PublishDelta: false
|
||||
Download:
|
||||
FilenameFormat: "trivy-db-{runId}.tar.gz"
|
||||
IncludeMetadata: true
|
||||
```
|
||||
|
||||
- `AllowEmpty=false` converts empty datasets into `ERR_EXPORT_EMPTY`.
|
||||
- `MaxCvssVectorsPerEntry` prevents extremely large multi-vector advisories from bloating the DB.
|
||||
- `PublishDelta` works in tandem with the planner's delta logic; when true, only changed blobs are pushed.
|
||||
- `FilenameFormat` lets operators align downloads with existing mirror tooling.
|
||||
- `IncludeMetadata` toggles whether `metadata.json` is stored alongside the bundle in the staging directory for quick inspection.
|
||||
|
||||
## 7. Distribution guidelines
|
||||
|
||||
- **Download profile**: `db.bundle` placed under `/export/trivy/` and signed. Recommended filename `trivy-db-<runId>.tar.gz`.
|
||||
- **OCI push**: ORAS artifact with annotations:
|
||||
- `org.opencontainers.artifact.description=StellaOps Trivy DB`
|
||||
- `io.stella.export.profile=trivy:db`
|
||||
- `io.stella.export.run=<runId>`
|
||||
- `io.stella.export.schemaVersion=2`
|
||||
- **Offline Kit**: When `offlineBundle.includeTrivyDb=true`, the exporter copies the latest full bundle plus the last `N` deltas (configurable) with manifests for quick import.
|
||||
|
||||
Consumers should always verify signatures using `trivy-db.sig` / `trivy-java-db.sig` before trusting the bundle.
|
||||
|
||||
Example verification flow:
|
||||
|
||||
```bash
|
||||
cosign verify-blob \
|
||||
--key tenants/acme/export-center.pub \
|
||||
--signature signatures/trivy-db.sig \
|
||||
trivy/db.bundle
|
||||
|
||||
trivy module db import trivy/db.bundle --cache-dir /tmp/trivy-cache
|
||||
```
|
||||
|
||||
## 8. Troubleshooting
|
||||
|
||||
| Symptom | Likely cause | Remedy |
|
||||
|---------|--------------|--------|
|
||||
| `ERR_EXPORT_UNSUPPORTED_SCHEMA` | Trivy CLI updated schema version. | Bump `SchemaVersion`, extend mapping tables, regenerate fixtures. |
|
||||
| `adapter.trivy.unsupported_namespace` | Advisory namespace not in allowlist. | Extend namespace mapping or exclude in selector. |
|
||||
| `trivy import` fails with "invalid bolt page" | Corrupted bundle or truncated upload. | Re-run export; verify storage backend and signatures. |
|
||||
| Missing Java advisories | `IncludeJavaDb=false` or no Java data in Findings Ledger. | Enable flag and confirm upstream connectors populate Java ecosystems. |
|
||||
| Severity downgraded to UNKNOWN | Source severity missing or unrecognized. | Ensure upstream connectors populate severity or supply CVSS scores. |
|
||||
| `ERR_EXPORT_EMPTY` returned unexpectedly | Selectors yielded zero records while `AllowEmpty=false`. | Review selectors; set `AllowEmpty=true` if empty exports are acceptable. |
|
||||
|
||||
## 9. References
|
||||
|
||||
- [Export Center API reference](api.md)
|
||||
- [Export Center CLI Guide](cli.md)
|
||||
- [Export Center Architecture](architecture.md)
|
||||
- [Export Center Overview](overview.md)
|
||||
- [Aqua Security Trivy documentation](https://aquasecurity.github.io/trivy/dev/database/structure/) *(external reference for schema expectations)*
|
||||
|
||||
> **Imposed rule:** Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.
|
||||
# Export Center Trivy Adapters
|
||||
|
||||
The Trivy adapters translate StellaOps normalized advisories into the format consumed by Aqua Security's Trivy scanner. They enable downstream tooling to reuse StellaOps' curated data without bespoke converters, while preserving Aggregation-Only Contract (AOC) boundaries. This guide documents bundle layouts, field mappings, compatibility guarantees, validation workflows, and configuration toggles introduced in Sprint 36 (`EXPORT-SVC-36-001`, `EXPORT-SVC-36-002`).
|
||||
|
||||
> The current Export Center build is wiring the API and workers. Treat this document as the canonical interface for adapter implementation and update any behavioural changes during task sign-off.
|
||||
|
||||
## 1. Adapter overview
|
||||
|
||||
| Variant | Bundle | Default profile | Notes |
|
||||
|---------|--------|-----------------|-------|
|
||||
| `trivy:db` | `db.bundle` | `trivy:db` | Core vulnerability database compatible with Trivy CLI >= 0.50.0 (schema v2). |
|
||||
| `trivy:java-db` | `java-db.bundle` | Optional extension | Java ecosystem supplement (Maven, Gradle). Enabled when `ExportCenter:Profiles:Trivy:EnableJavaDb=true`. |
|
||||
|
||||
Both variants ship inside the export run under `/export/trivy/`. Each bundle is a gzip-compressed tarball containing:
|
||||
|
||||
```
|
||||
metadata.json
|
||||
trivy.db # BoltDB file with vulnerability/provider tables
|
||||
packages/*.json # Only when schema requires JSON overlays (language ecosystems)
|
||||
```
|
||||
|
||||
The adapters never mutate input evidence. They only reshape normalized advisories and copy the exact upstream references so consumers can trace provenance.
|
||||
|
||||
## 2. Bundle layout
|
||||
|
||||
```
|
||||
trivy/
|
||||
db.bundle
|
||||
+-- metadata.json
|
||||
+-- trivy.db
|
||||
java-db.bundle # present when Java DB enabled
|
||||
+-- metadata.json
|
||||
+-- trivy-java.db
|
||||
+-- ecosystem/...
|
||||
signatures/
|
||||
trivy-db.sig
|
||||
trivy-java-db.sig
|
||||
```
|
||||
|
||||
`metadata.json` aligns with Trivy's expectations (`schemaVersion`, `buildInfo`, `updatedAt`, etc.). Export Center adds an `stella` block to capture profile id, run id, and policy snapshot hints.
|
||||
|
||||
Example `metadata.json` (trimmed):
|
||||
|
||||
```json
|
||||
{
|
||||
"schemaVersion": 2,
|
||||
"buildInfo": {
|
||||
"trivyVersion": "0.50.1",
|
||||
"vulnerabilityDBVersion": "2025-10-28T00:00:00Z"
|
||||
},
|
||||
"updatedAt": "2025-10-29T11:42:03Z",
|
||||
"stella": {
|
||||
"runId": "run-20251029-01",
|
||||
"profileId": "prof-trivy-db",
|
||||
"tenant": "acme",
|
||||
"policySnapshotId": "policy-snap-42",
|
||||
"schemaVersion": 2
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## 3. Field mappings
|
||||
|
||||
### 3.1 Namespace resolution
|
||||
|
||||
| Stella field | Trivy field | Notes |
|
||||
|--------------|-------------|-------|
|
||||
| `advisory.source.vendor` | `namespace` | Canonicalized to lowercase; e.g. `Ubuntu` -> `ubuntu`. |
|
||||
| `advisory.source.product` | `distribution` / `ecosystem` | Mapped via allowlist (`Ubuntu 22.04` -> `ubuntu:22.04`). |
|
||||
| `package.ecosystem` | `package.ecosystem` | OSS ecosystems (`npm`, `pip`, `nuget`, etc.). |
|
||||
| `package.nevra` / `package.evr` | `package.version` (OS) | RPM/DEB version semantics preserved. |
|
||||
|
||||
If a record lacks a supported namespace, the adapter drops it and logs `adapter.trivy.unsupported_namespace`.
|
||||
|
||||
### 3.2 Vulnerability metadata
|
||||
|
||||
| Stella field | Trivy field | Transformation |
|
||||
|--------------|-------------|----------------|
|
||||
| `advisory.identifiers.cve[]` | `vulnerability.CVEIDs` | Array of strings. |
|
||||
| `advisory.identifiers.aliases[]` | `vulnerability.CWEIDs` / `References` | CVE -> `CVEIDs`, others appended to `References`. |
|
||||
| `advisory.summary` | `vulnerability.Title` | Stripped to 256 chars; rest moved to `Description`. |
|
||||
| `advisory.description` | `vulnerability.Description` | Markdown allowed, normalized to LF line endings. |
|
||||
| `advisory.severity.normalized` | `vulnerability.Severity` | Uses table below. |
|
||||
| `advisory.cvss[]` | `vulnerability.CVSS` | Stored as `{"vector": "...", "score": 7.8, "source": "NVD"}`. |
|
||||
| `advisory.published` | `vulnerability.PublishedDate` | ISO 8601 UTC. |
|
||||
| `advisory.modified` | `vulnerability.LastModifiedDate` | ISO 8601 UTC. |
|
||||
| `advisory.vendorStatement` | `vulnerability.VendorSeverity` / `VendorVectors` | Preserved in vendor block. |
|
||||
|
||||
Severity mapping:
|
||||
|
||||
| Stella severity | Trivy severity |
|
||||
|-----------------|----------------|
|
||||
| `critical` | `CRITICAL` |
|
||||
| `high` | `HIGH` |
|
||||
| `medium` | `MEDIUM` |
|
||||
| `low` | `LOW` |
|
||||
| `none` / `info` | `UNKNOWN` |
|
||||
|
||||
### 3.3 Affected packages
|
||||
|
||||
| Stella field | Trivy field | Notes |
|
||||
|--------------|-------------|-------|
|
||||
| `package.name` | `package.name` | For OS distros uses source package when available. |
|
||||
| `package.purl` | `package.PURL` | Copied verbatim. |
|
||||
| `affects.vulnerableRange` | `package.vulnerableVersionRange` | SemVer or distro version range. |
|
||||
| `remediations.fixedVersion` | `package.fixedVersion` | Latest known fix. |
|
||||
| `remediations.urls[]` | `package.links` | Array; duplicates removed. |
|
||||
| `states.cpes[]` | `package.cpes` | For CPE-backed advisories. |
|
||||
|
||||
The adapter deduplicates entries by `(namespace, package.name, vulnerableRange)` to avoid duplicate records when multiple upstream segments agree.
|
||||
|
||||
Example mapping (Ubuntu advisory):
|
||||
|
||||
```jsonc
|
||||
// Stella normalized input
|
||||
{
|
||||
"source": {"vendor": "Ubuntu", "product": "22.04"},
|
||||
"identifiers": {"cve": ["CVE-2024-12345"]},
|
||||
"severity": {"normalized": "high"},
|
||||
"affects": [{
|
||||
"package": {"name": "openssl", "ecosystem": "ubuntu", "nevra": "1.1.1f-1ubuntu2.12"},
|
||||
"vulnerableRange": "< 1.1.1f-1ubuntu2.13",
|
||||
"remediations": [{"fixedVersion": "1.1.1f-1ubuntu2.13"}]
|
||||
}]
|
||||
}
|
||||
|
||||
// Trivy vulnerability entry
|
||||
{
|
||||
"namespace": "ubuntu",
|
||||
"package": {
|
||||
"name": "openssl",
|
||||
"version": "< 1.1.1f-1ubuntu2.13",
|
||||
"fixedVersion": "1.1.1f-1ubuntu2.13"
|
||||
},
|
||||
"vulnerability": {
|
||||
"ID": "CVE-2024-12345",
|
||||
"Severity": "HIGH"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 3.4 Java DB specifics
|
||||
|
||||
The Java supplement only includes ecosystems `maven`, `gradle`, `sbt`. Additional fields:
|
||||
|
||||
| Stella field | Trivy Java field | Notes |
|
||||
|--------------|------------------|-------|
|
||||
| `package.group` | `GroupID` | Derived from Maven coordinates. |
|
||||
| `package.artifact` | `ArtifactID` | Derived from Maven coordinates. |
|
||||
| `package.version` | `Version` | Compared with semver-lite rules. |
|
||||
| `affects.symbolicRanges[]` | `VulnerableVersions` | Strings like `[1.0.0,1.2.3)`. |
|
||||
|
||||
## 4. Compatibility matrix
|
||||
|
||||
| Trivy version | Schema version | Supported by adapter | Notes |
|
||||
|---------------|----------------|----------------------|-------|
|
||||
| 0.46.x | 2 | Yes | Baseline compatibility target. |
|
||||
| 0.50.x | 2 | Yes | Default validation target in CI. |
|
||||
| 0.51.x+ | 3 | Pending | Adapter throws `ERR_EXPORT_UNSUPPORTED_SCHEMA` until implemented. |
|
||||
|
||||
Schema mismatches emit `adapter.trivy.unsupported_schema_version` and abort the run. Operators can pin the schema via `ExportCenter:Adapters:Trivy:SchemaVersion`.
|
||||
|
||||
## 5. Validation workflow
|
||||
|
||||
1. **Unit tests** (`StellaOps.ExportCenter.Tests`):
|
||||
- Mapping tests for OS and ecosystem packages.
|
||||
- Severity conversion and range handling property tests.
|
||||
2. **Integration tests** (`EXPORT-SVC-36-001`):
|
||||
- Generate bundle from fixture dataset.
|
||||
- Run `trivy module db import <bundle>` (Trivy CLI) to ensure the bundle is accepted.
|
||||
- For Java DB, run `trivy java-repo --db <bundle>` against sample repository.
|
||||
3. **CI smoke (`DEVOPS-EXPORT-36-001`)**:
|
||||
- Validate metadata fields using `jq`.
|
||||
- Ensure signatures verify with `cosign`.
|
||||
- Check runtime by invoking `trivy fs --cache-dir <temp> --skip-update --custom-db <bundle> fixtures/image`.
|
||||
|
||||
Failures set the run status to `failed` with `errorCode="adapter-trivy"` so Console/CLI expose the reason.
|
||||
|
||||
## 6. Configuration knobs
|
||||
|
||||
```yaml
|
||||
ExportCenter:
|
||||
Adapters:
|
||||
Trivy:
|
||||
SchemaVersion: 2 # enforce schema version
|
||||
IncludeJavaDb: true # enable java-db.bundle
|
||||
AllowEmpty: false # fail when no records match
|
||||
MaxCvssVectorsPerEntry: 5 # truncate to avoid oversized payloads
|
||||
Distribution:
|
||||
Oras:
|
||||
TrivyRepository: "registry.example.com/stella/trivy-db"
|
||||
PublishDelta: false
|
||||
Download:
|
||||
FilenameFormat: "trivy-db-{runId}.tar.gz"
|
||||
IncludeMetadata: true
|
||||
```
|
||||
|
||||
- `AllowEmpty=false` converts empty datasets into `ERR_EXPORT_EMPTY`.
|
||||
- `MaxCvssVectorsPerEntry` prevents extremely large multi-vector advisories from bloating the DB.
|
||||
- `PublishDelta` works in tandem with the planner's delta logic; when true, only changed blobs are pushed.
|
||||
- `FilenameFormat` lets operators align downloads with existing mirror tooling.
|
||||
- `IncludeMetadata` toggles whether `metadata.json` is stored alongside the bundle in the staging directory for quick inspection.
|
||||
|
||||
## 7. Distribution guidelines
|
||||
|
||||
- **Download profile**: `db.bundle` placed under `/export/trivy/` and signed. Recommended filename `trivy-db-<runId>.tar.gz`.
|
||||
- **OCI push**: ORAS artifact with annotations:
|
||||
- `org.opencontainers.artifact.description=StellaOps Trivy DB`
|
||||
- `io.stella.export.profile=trivy:db`
|
||||
- `io.stella.export.run=<runId>`
|
||||
- `io.stella.export.schemaVersion=2`
|
||||
- **Offline Kit**: When `offlineBundle.includeTrivyDb=true`, the exporter copies the latest full bundle plus the last `N` deltas (configurable) with manifests for quick import.
|
||||
|
||||
Consumers should always verify signatures using `trivy-db.sig` / `trivy-java-db.sig` before trusting the bundle.
|
||||
|
||||
Example verification flow:
|
||||
|
||||
```bash
|
||||
cosign verify-blob \
|
||||
--key tenants/acme/export-center.pub \
|
||||
--signature signatures/trivy-db.sig \
|
||||
trivy/db.bundle
|
||||
|
||||
trivy module db import trivy/db.bundle --cache-dir /tmp/trivy-cache
|
||||
```
|
||||
|
||||
## 8. Troubleshooting
|
||||
|
||||
| Symptom | Likely cause | Remedy |
|
||||
|---------|--------------|--------|
|
||||
| `ERR_EXPORT_UNSUPPORTED_SCHEMA` | Trivy CLI updated schema version. | Bump `SchemaVersion`, extend mapping tables, regenerate fixtures. |
|
||||
| `adapter.trivy.unsupported_namespace` | Advisory namespace not in allowlist. | Extend namespace mapping or exclude in selector. |
|
||||
| `trivy import` fails with "invalid bolt page" | Corrupted bundle or truncated upload. | Re-run export; verify storage backend and signatures. |
|
||||
| Missing Java advisories | `IncludeJavaDb=false` or no Java data in Findings Ledger. | Enable flag and confirm upstream connectors populate Java ecosystems. |
|
||||
| Severity downgraded to UNKNOWN | Source severity missing or unrecognized. | Ensure upstream connectors populate severity or supply CVSS scores. |
|
||||
| `ERR_EXPORT_EMPTY` returned unexpectedly | Selectors yielded zero records while `AllowEmpty=false`. | Review selectors; set `AllowEmpty=true` if empty exports are acceptable. |
|
||||
|
||||
## 9. References
|
||||
|
||||
- [Export Center API reference](api.md)
|
||||
- [Export Center CLI Guide](cli.md)
|
||||
- [Export Center Architecture](architecture.md)
|
||||
- [Export Center Overview](overview.md)
|
||||
- [Aqua Security Trivy documentation](https://aquasecurity.github.io/trivy/dev/database/structure/) *(external reference for schema expectations)*
|
||||
|
||||
> **Imposed rule:** Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.
|
||||
|
||||
@@ -1,96 +1,96 @@
|
||||
# Policy Engine FAQ
|
||||
|
||||
Answers to questions that Support, Ops, and Policy Guild teams receive most frequently. Pair this FAQ with the [Policy Lifecycle](../policy/lifecycle.md), [Runs](../policy/runs.md), and [CLI guide](../cli/policy.md) for deeper explanations.
|
||||
|
||||
---
|
||||
|
||||
## Authoring & DSL
|
||||
|
||||
**Q:** *Lint succeeds locally, but submit still fails with `ERR_POL_001`. Why?*
|
||||
**A:** The CLI requires lint & compile artefacts newer than 24 hours. Re-run `stella policy lint` and `stella policy compile` before submitting; ensure you upload the latest diff files with `--attach`.
|
||||
|
||||
**Q:** *How do I layer tenant-specific overrides on top of the baseline policy?*
|
||||
**A:** Keep the baseline in `tenant-global`. For tenant overrides, create a policy referencing the baseline via CLI (`stella policy new --from baseline@<version>`), then adjust rules. Activation is per tenant.
|
||||
|
||||
**Q:** *Can I import YAML/Rego policies from earlier releases?*
|
||||
**A:** No direct import. Use the migration script (`stella policy migrate legacy.yaml`) which outputs `stella-dsl@1` skeletons. Review manually before submission.
|
||||
|
||||
---
|
||||
|
||||
## Simulation & Determinism
|
||||
|
||||
**Q:** *Simulation shows huge differences even though I only tweaked metadata. What did I miss?*
|
||||
**A:** Check if your simulation used the same SBOM set/env as previous runs. CLI default uses golden fixtures; UI can store custom presets. Large diffs may also indicate Concelier updates; compare advisory cursors in the Simulation tab.
|
||||
|
||||
**Q:** *How do we guard against non-deterministic behaviour?*
|
||||
**A:** CI runs `policy simulate` twice with identical inputs and compares outputs (`DEVOPS-POLICY-20-003`). Any difference fails the pipeline. Locally you can use `stella policy run replay` to verify determinism.
|
||||
|
||||
**Q:** *What happens if the determinism guard (`ERR_POL_004`) triggers?*
|
||||
**A:** Policy Engine halts the run, raises `policy.run.failed` with code `ERR_POL_004`, and switches to incident mode (100 % sampling). Review recent code changes; often caused by new helpers that call `DateTime.Now` or non-allowlisted HTTP clients.
|
||||
|
||||
---
|
||||
|
||||
## VEX & Suppressions
|
||||
|
||||
**Q:** *A vendor marked a CVE `not_affected` but the policy still blocks. Why?*
|
||||
**A:** Check the required justifications. Baseline policy only accepts `component_not_present` and `vulnerable_code_not_present`. Other statuses need explicit rules. Use `stella findings explain` to see which VEX statement was considered.
|
||||
|
||||
**Q:** *Can we quiet a finding indefinitely?*
|
||||
**A:** Avoid indefinite quiets. Policy DSL requires an `until` timestamp. If the use case is permanent, move the rule into baseline logic with strong justification and documentation.
|
||||
|
||||
**Q:** *How do we detect overuse of suppressions?*
|
||||
**A:** Observability exports `policy_suppressions_total` and CLI `stella policy stats`. Review weekly; Support flags tenants whose suppressions grow faster than remediation tickets.
|
||||
|
||||
---
|
||||
|
||||
## Runs & Operations
|
||||
|
||||
**Q:** *Incremental runs are backlogged. What should we check first?*
|
||||
**A:** Inspect `policy_run_queue_depth` and `policy_delta_backlog_age_seconds` dashboards. If queue depth high, scale worker replicas or investigate upstream change storms (Concelier/Excititor). Use `stella policy run list --status failed` for recent errors.
|
||||
|
||||
**Q:** *Full runs take longer than 30 min. Is that a breach?*
|
||||
**A:** Goal is ≤ 30 min, but large tenants may exceed temporarily. Ensure Mongo indexes are current and that worker nodes meet sizing (4 vCPU). Consider sharding runs by SBOM group.
|
||||
|
||||
**Q:** *How do I replay a run for audit evidence?*
|
||||
**A:** `stella policy run replay <runId> --output replay.tgz` produces a sealed bundle. Upload to evidence locker or attach to incident tickets.
|
||||
|
||||
---
|
||||
|
||||
## Approvals & Governance
|
||||
|
||||
**Q:** *Can authors approve their own policies?*
|
||||
**A:** No. Authority denies approval if `approved_by == submitted_by`. Assign at least two reviewers (one security, one product).
|
||||
|
||||
**Q:** *What scopes do bots need for CI pipelines?*
|
||||
**A:** Typically `policy:read`, `policy:simulate`, `policy:runs`. Only grant `policy:run` if the pipeline should trigger runs. Never give CI tokens `policy:approve`.
|
||||
|
||||
**Q:** *How do we manage policies in air-gapped deployments?*
|
||||
**A:** Use `stella policy bundle export --sealed` on a connected site, transfer via approved media, then `stella policy bundle import` inside the enclave. Enable `--sealed` flag in CLI/UI to block accidental outbound calls.
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
**Q:** *API calls return `403` despite valid token.*
|
||||
**A:** Verify scope includes the specific operation (`policy:activate` vs `policy:run`). Check tenant header matches token tenant. Inspect Authority logs for denial reason (`policy_scope_denied_total` metric).
|
||||
|
||||
**Q:** *`stella policy run` exits with code `30`.*
|
||||
**A:** Network/transport error. Check connectivity to Policy Engine endpoint, TLS configuration, and CLI proxy settings.
|
||||
|
||||
**Q:** *Explain drawer shows no VEX data.*
|
||||
**A:** Either no VEX statement matched or the tenant lacks `findings:read` scope. If VEX should exist, confirm Excititor ingestion and policy joiners (see Observability dashboards).
|
||||
|
||||
---
|
||||
|
||||
## Compliance Checklist
|
||||
|
||||
- [ ] FAQ linked from Console help menu and CLI `stella policy help`.
|
||||
- [ ] Entries reviewed quarterly by Policy & Support Guilds.
|
||||
- [ ] Answers cross-reference lifecycle, runs, observability, and governance docs.
|
||||
- [ ] Incident/Escalation contact details kept current in Support playbooks.
|
||||
- [ ] FAQ translated for supported locales (if applicable).
|
||||
|
||||
---
|
||||
|
||||
*Last updated: 2025-10-26 (Sprint 20).*
|
||||
|
||||
# Policy Engine FAQ
|
||||
|
||||
Answers to questions that Support, Ops, and Policy Guild teams receive most frequently. Pair this FAQ with the [Policy Lifecycle](../policy/lifecycle.md), [Runs](../policy/runs.md), and [CLI guide](../cli/policy.md) for deeper explanations.
|
||||
|
||||
---
|
||||
|
||||
## Authoring & DSL
|
||||
|
||||
**Q:** *Lint succeeds locally, but submit still fails with `ERR_POL_001`. Why?*
|
||||
**A:** The CLI requires lint & compile artefacts newer than 24 hours. Re-run `stella policy lint` and `stella policy compile` before submitting; ensure you upload the latest diff files with `--attach`.
|
||||
|
||||
**Q:** *How do I layer tenant-specific overrides on top of the baseline policy?*
|
||||
**A:** Keep the baseline in `tenant-global`. For tenant overrides, create a policy referencing the baseline via CLI (`stella policy new --from baseline@<version>`), then adjust rules. Activation is per tenant.
|
||||
|
||||
**Q:** *Can I import YAML/Rego policies from earlier releases?*
|
||||
**A:** No direct import. Use the migration script (`stella policy migrate legacy.yaml`) which outputs `stella-dsl@1` skeletons. Review manually before submission.
|
||||
|
||||
---
|
||||
|
||||
## Simulation & Determinism
|
||||
|
||||
**Q:** *Simulation shows huge differences even though I only tweaked metadata. What did I miss?*
|
||||
**A:** Check if your simulation used the same SBOM set/env as previous runs. CLI default uses golden fixtures; UI can store custom presets. Large diffs may also indicate Concelier updates; compare advisory cursors in the Simulation tab.
|
||||
|
||||
**Q:** *How do we guard against non-deterministic behaviour?*
|
||||
**A:** CI runs `policy simulate` twice with identical inputs and compares outputs (`DEVOPS-POLICY-20-003`). Any difference fails the pipeline. Locally you can use `stella policy run replay` to verify determinism.
|
||||
|
||||
**Q:** *What happens if the determinism guard (`ERR_POL_004`) triggers?*
|
||||
**A:** Policy Engine halts the run, raises `policy.run.failed` with code `ERR_POL_004`, and switches to incident mode (100 % sampling). Review recent code changes; often caused by new helpers that call `DateTime.Now` or non-allowlisted HTTP clients.
|
||||
|
||||
---
|
||||
|
||||
## VEX & Suppressions
|
||||
|
||||
**Q:** *A vendor marked a CVE `not_affected` but the policy still blocks. Why?*
|
||||
**A:** Check the required justifications. Baseline policy only accepts `component_not_present` and `vulnerable_code_not_present`. Other statuses need explicit rules. Use `stella findings explain` to see which VEX statement was considered.
|
||||
|
||||
**Q:** *Can we quiet a finding indefinitely?*
|
||||
**A:** Avoid indefinite quiets. Policy DSL requires an `until` timestamp. If the use case is permanent, move the rule into baseline logic with strong justification and documentation.
|
||||
|
||||
**Q:** *How do we detect overuse of suppressions?*
|
||||
**A:** Observability exports `policy_suppressions_total` and CLI `stella policy stats`. Review weekly; Support flags tenants whose suppressions grow faster than remediation tickets.
|
||||
|
||||
---
|
||||
|
||||
## Runs & Operations
|
||||
|
||||
**Q:** *Incremental runs are backlogged. What should we check first?*
|
||||
**A:** Inspect `policy_run_queue_depth` and `policy_delta_backlog_age_seconds` dashboards. If queue depth high, scale worker replicas or investigate upstream change storms (Concelier/Excititor). Use `stella policy run list --status failed` for recent errors.
|
||||
|
||||
**Q:** *Full runs take longer than 30 min. Is that a breach?*
|
||||
**A:** Goal is ≤ 30 min, but large tenants may exceed temporarily. Ensure Mongo indexes are current and that worker nodes meet sizing (4 vCPU). Consider sharding runs by SBOM group.
|
||||
|
||||
**Q:** *How do I replay a run for audit evidence?*
|
||||
**A:** `stella policy run replay <runId> --output replay.tgz` produces a sealed bundle. Upload to evidence locker or attach to incident tickets.
|
||||
|
||||
---
|
||||
|
||||
## Approvals & Governance
|
||||
|
||||
**Q:** *Can authors approve their own policies?*
|
||||
**A:** No. Authority denies approval if `approved_by == submitted_by`. Assign at least two reviewers (one security, one product).
|
||||
|
||||
**Q:** *What scopes do bots need for CI pipelines?*
|
||||
**A:** Typically `policy:read`, `policy:simulate`, `policy:runs`. Only grant `policy:run` if the pipeline should trigger runs. Never give CI tokens `policy:approve`.
|
||||
|
||||
**Q:** *How do we manage policies in air-gapped deployments?*
|
||||
**A:** Use `stella policy bundle export --sealed` on a connected site, transfer via approved media, then `stella policy bundle import` inside the enclave. Enable `--sealed` flag in CLI/UI to block accidental outbound calls.
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
**Q:** *API calls return `403` despite valid token.*
|
||||
**A:** Verify scope includes the specific operation (`policy:activate` vs `policy:run`). Check tenant header matches token tenant. Inspect Authority logs for denial reason (`policy_scope_denied_total` metric).
|
||||
|
||||
**Q:** *`stella policy run` exits with code `30`.*
|
||||
**A:** Network/transport error. Check connectivity to Policy Engine endpoint, TLS configuration, and CLI proxy settings.
|
||||
|
||||
**Q:** *Explain drawer shows no VEX data.*
|
||||
**A:** Either no VEX statement matched or the tenant lacks `findings:read` scope. If VEX should exist, confirm Excititor ingestion and policy joiners (see Observability dashboards).
|
||||
|
||||
---
|
||||
|
||||
## Compliance Checklist
|
||||
|
||||
- [ ] FAQ linked from Console help menu and CLI `stella policy help`.
|
||||
- [ ] Entries reviewed quarterly by Policy & Support Guilds.
|
||||
- [ ] Answers cross-reference lifecycle, runs, observability, and governance docs.
|
||||
- [ ] Incident/Escalation contact details kept current in Support playbooks.
|
||||
- [ ] FAQ translated for supported locales (if applicable).
|
||||
|
||||
---
|
||||
|
||||
*Last updated: 2025-10-26 (Sprint 20).*
|
||||
|
||||
|
||||
112
docs/implplan/AGENTS.md
Normal file
112
docs/implplan/AGENTS.md
Normal file
@@ -0,0 +1,112 @@
|
||||
# 1) What is StellaOps?
|
||||
|
||||
**StellaOps** an open, sovereign, modular container-security toolkit built for high-speed, offline operation, released under AGPL-3.0-or-later.
|
||||
|
||||
It follows an SBOM-first model—analyzing each container layer or ingesting existing CycloneDX/SPDX SBOMs, then enriching them with vulnerability, licence, secret-leak, and misconfiguration data to produce cryptographically signed reports.
|
||||
|
||||
Vulnerability detection maps OS and language dependencies to sources such as NVD, GHSA, OSV, ENISA.
|
||||
Secrets sweep flags exposed credentials or keys in files or environment variables.
|
||||
Licence audit identifies potential conflicts, especially copyleft obligations.
|
||||
Misconfiguration checks detect unsafe Dockerfile patterns (root user, latest tags, permissive modes).
|
||||
Provenance features include in-toto/SLSA attestations signed with cosign for supply-chain trust.
|
||||
|
||||
| Guiding principle | What it means for Concelier |
|
||||
|-------------------|---------------------------|
|
||||
| **SBOM-first ingest** | Prefer signed SBOMs or reproducible layer diffs before falling back to raw scraping; connectors treat source docs as provenance, never as mutable truth. |
|
||||
| **Deterministic outputs** | Same inputs yield identical canonical advisories and exported JSON/Trivy DB artefacts; merge hashes and export manifests are reproducible across machines. |
|
||||
| **Restart-time plug-ins only** | Connector/exporter plug-ins load at service start, keeping runtime sandboxing simple and avoiding hot-patch attack surface. |
|
||||
| **Sovereign/offline-first** | No mandatory outbound calls beyond allow-listed advisories; Offline Kit bundles Mongo snapshots and exporter artefacts for air-gapped installs. |
|
||||
| **Operational transparency** | Every stage logs structured events (fetch, parse, merge, export) with correlation IDs so parallel agents can debug without shared state. |
|
||||
|
||||
Performance: warm scans < 5 s, cold scans < 30 s on a 4 vCPU runner.
|
||||
Deployment: entirely SaaS-free, suitable for air-gapped or on-prem use through its Offline Kit.
|
||||
Policy: anonymous users → 33 scans/day; verified → 333 /day; nearing 90 % quota triggers throttling but never full blocks.
|
||||
|
||||
More documention is available ./docs/*.md files. Read `docs/README.md` to gather information about the available documentation. You could inquiry specific documents as your work requires it
|
||||
|
||||
---
|
||||
|
||||
# 3) Practices
|
||||
|
||||
## 3.1) Naming
|
||||
All modules are .NET projects based on .NET 10 (preview). Exclussion is the UI. It is based on Angular
|
||||
All modules are contained by one or more projects. Each project goes in its dedicated folder. Each project starts with StellaOps.<ModuleName>. In case it is common for for all StellaOps modules it is library or plugin and it is named StellaOps.<LibraryOrPlugin>.
|
||||
|
||||
## 3.2) Key technologies & integrations
|
||||
|
||||
- **Runtime**: .NET 10 (`net10.0`) preview SDK; C# latest preview features.
|
||||
- **Data**: MongoDB (canonical store and job/export state).
|
||||
- **Observability**: structured logs, counters, and (optional) OpenTelemetry traces.
|
||||
- **Ops posture**: offline‑first, allowlist for remote hosts, strict schema validation, gated LLM fallback (only where explicitly configured).
|
||||
|
||||
# 4) Modules
|
||||
StellaOps is contained by different modules installable via docker containers
|
||||
- Concelier. Responsible for aggregation and delivery of vulnerability database
|
||||
- Cli. Command line tool to unlock full potential - request database operations, install scanner, request scan, configure backend
|
||||
- Backend. Configures and Manages scans
|
||||
- UI. UI to access the backend (and scanners)
|
||||
- Agent. Installable daemon that does the scanning
|
||||
- Zastava. Realtime monitor for allowed (verified) installations.
|
||||
|
||||
For more information of the architecture see `./docs/*ARCHITECTURE*.md` files.
|
||||
|
||||
---
|
||||
|
||||
### 4.1.4) Glossary (quick)
|
||||
|
||||
- **OVAL** — Vendor/distro security definition format; authoritative for OS packages.
|
||||
- **NEVRA / EVR** — RPM and Debian version semantics for OS packages.
|
||||
- **PURL / SemVer** — Coordinates and version semantics for OSS ecosystems.
|
||||
- **KEV** — Known Exploited Vulnerabilities (flag only).
|
||||
|
||||
---
|
||||
# 5) Your role as StellaOps contributor
|
||||
|
||||
You acting as information technology engineer that will take different type of roles in goal achieving StellaOps production implementation
|
||||
In order you to work - you have to be supplied with directory that contains `AGENTS.md`,`TASKS.md` files. There will you have more information about the role you have, the scope of your work and the tasks you will have.
|
||||
|
||||
Boundaries:
|
||||
- You operate only in the working directories I gave you, unless there is dependencies that makes you to work on dependency in shared directory. Then you ask for confirmation.
|
||||
|
||||
You main characteristics:
|
||||
- Keep endpoints small, deterministic, and cancellation-aware.
|
||||
- Improve logs/metrics as per tasks.
|
||||
- Update `TASKS.md` when moving tasks forward.
|
||||
- When you are done with all task you state explicitly you are done.
|
||||
- Impersonate the role described on working directory `AGENTS.md` you will read, if role is not available - take role of the CTO of the StellaOps in early stages.
|
||||
- You always strive for best practices
|
||||
- You always strive for re-usability
|
||||
- When in doubt of design decision - you ask then act
|
||||
- You are autonomus - meaning that you will work for long time alone and achieve maximum without stopping for stupid questions
|
||||
- You operate on the same directory where other agents will work. In case you need to work on directory that is dependency on provided `AGENTS.md`,`TASKS.md` files you have to ask for confirmation first.
|
||||
|
||||
## 5.1) Type of contributions
|
||||
|
||||
- **BE‑Base (Platform & Pipeline)**
|
||||
Owns DI, plugin host, job scheduler/coordinator, configuration binding, minimal API endpoints, and Mongo bootstrapping.
|
||||
- **BE‑Conn‑X (Connectors)**
|
||||
One agent per source family (NVD, Red Hat, Ubuntu, Debian, SUSE, GHSA, OSV, PSIRTs, CERTs, KEV, ICS). Implements fetch/parse/map with incremental watermarks.
|
||||
- **BE‑Merge (Canonical Merge & Dedupe)**
|
||||
Identity graph, precedence policies, canonical JSON serializer, and deterministic hashing (`merge_event`).
|
||||
- **BE‑Export (JSON & Trivy DB)**
|
||||
Deterministic export trees, Trivy DB packaging, optional ORAS push, and offline bundle.
|
||||
- **QA (Validation & Observability)**
|
||||
Schema tests, fixture goldens, determinism checks, metrics/logs/traces, e2e reproducibility runs.
|
||||
- **DevEx/Docs**
|
||||
Maintains this agent framework, templates, and per‑directory guides; assists parallelization and reviews.
|
||||
|
||||
|
||||
## 5.2) Work rules (important)
|
||||
|
||||
- **Directory ownership**: Each agent works **only inside its module directory**. Cross‑module edits require a brief handshake in issues/PR description.
|
||||
- **Scoping**: Use each module’s `AGENTS.md` and `TASKS.md` to plan; autonomous agents must read `src/AGENTS.md` and the module docs before acting.
|
||||
- **Determinism**: Sort keys, normalize timestamps to UTC ISO‑8601, avoid non‑deterministic data in exports and tests.
|
||||
- **Status tracking**: Update your module’s `TASKS.md` as you progress (TODO → DOING → DONE/BLOCKED). Before starting of actual work - ensure you have set the task to DOING. When complete or stop update the status in corresponding TASKS.md and in ./SPRINTS.md file.
|
||||
- **Coordination**: In case task is discovered as blocked on other team or task, according TASKS.md files that dependency is on needs to be changed by adding new tasks describing the requirement. the current task must be updated as completed. In case task changes, scope or requirements or rules - other documentations needs be updated accordingly.
|
||||
- **Sprint synchronization**: When given task seek for relevant directory to work on from SPRINTS.md. Confirm its state on both SPRINTS.md and the relevant TASKS.md file. Always check the AGENTS.md in the relevant TASKS.md directory.
|
||||
- **Tests**: Add/extend fixtures and unit tests per change; never regress determinism or precedence.
|
||||
- **Test layout**: Use module-specific projects in `StellaOps.Concelier.<Component>.Tests`; shared fixtures/harnesses live in `StellaOps.Concelier.Testing`.
|
||||
- **Execution autonomous**: In case you need to continue with more than one options just continue sequentially, unless the continue requires design decision.
|
||||
- **EPIC reference**: In case task (on relevant TASKS.md file) references in direct or indirect way an EPIC - then seek for epic document like ./EPIC_*.md. There will be more information how to implement the task.
|
||||
|
||||
---
|
||||
524
docs/implplan/EPIC_1.md
Normal file
524
docs/implplan/EPIC_1.md
Normal file
@@ -0,0 +1,524 @@
|
||||
Here’s the full write‑up you can drop into your repo as the canonical reference for Epic 1. It’s written in clean product‑doc style so it’s safe to check in as Markdown. No fluff, just everything you need to build, test, and police it.
|
||||
|
||||
---
|
||||
|
||||
# Epic 1: Aggregation‑Only Contract (AOC) Enforcement
|
||||
|
||||
> Short name: **AOC enforcement**
|
||||
> Services touched: **Conseiller (advisory ingestion), Excitator (VEX ingestion), Web API, Workers, Policy Engine, CLI, Console, Authority**
|
||||
> Data stores: **MongoDB primary, optional Redis/NATS for jobs**
|
||||
|
||||
---
|
||||
|
||||
## 1) What it is
|
||||
|
||||
**Aggregation‑Only Contract (AOC)** is the ingestion covenant for StellaOps. It defines a hard boundary between **collection** and **interpretation**:
|
||||
|
||||
* **Ingestion (Conseiller/Excitator)** only **collects** data and preserves it as immutable raw facts with provenance. It does not decide, merge, normalize, prioritize, or assign severity. It may compute **links** that help future joins (aliases, PURLs, CPEs), but never derived judgments.
|
||||
* **Policy evaluation** is the only place where merges, deduplication, consensus, severity computation, and status folding are allowed. It’s reproducible and traceable.
|
||||
|
||||
The AOC establishes:
|
||||
|
||||
* **Immutable raw stores**: `advisory_raw` and `vex_raw` documents with full provenance, signatures, checksums, and upstream identifiers.
|
||||
* **Linksets**: machine‑generated join hints (aliases, PURLs, CPEs, CVE/GHSA IDs) that never change the underlying source content.
|
||||
* **Invariants**: a strict set of “never do this in ingestion” rules enforced by schema validation, runtime guards, and CI checks.
|
||||
* **AOC Verifier**: a build‑time and runtime watchdog that blocks non‑compliant code and data writes.
|
||||
|
||||
This epic delivers: schemas, guards, error codes, APIs, tests, migration, docs, and ops dashboards to make AOC non‑negotiable across the platform.
|
||||
|
||||
---
|
||||
|
||||
## 2) Why
|
||||
|
||||
AOC makes results **auditable, deterministic, and organization‑specific**. Source vendors disagree; your policies decide. By removing hidden heuristics from ingestion, we avoid unexplainable risk changes, race conditions between collectors, and vendor bias. Policy‑time evaluation yields reproducible deltas with complete “why” traces.
|
||||
|
||||
---
|
||||
|
||||
## 3) How it should work (deep details)
|
||||
|
||||
### 3.1 Core invariants
|
||||
|
||||
The following must be true for every write to `advisory_raw` and `vex_raw` and for every ingestion pipeline:
|
||||
|
||||
1. **No severity in ingestion**
|
||||
|
||||
* Forbidden fields: `severity`, `cvss`, `cvss_vector`, `effective_status`, `effective_range`, `merged_from`, `consensus_provider`, `reachability`, `asset_criticality`, `risk_score`.
|
||||
2. **No merges or de‑dups in ingestion**
|
||||
|
||||
* No combining two upstream advisories into one. No picking a single truth when multiple VEX statements exist.
|
||||
3. **Provenance is mandatory**
|
||||
|
||||
* Every raw doc includes `provenance` and `signature/checksum`.
|
||||
4. **Idempotent upserts**
|
||||
|
||||
* Same upstream document (by `upstream_id` + `source` + `content_hash`) must not create duplicates.
|
||||
5. **Append‑only versioning**
|
||||
|
||||
* Revisions from the source create new immutable documents with `supersedes` pointers; no in‑place edits.
|
||||
6. **Linkset only**
|
||||
|
||||
* Ingestion can compute and store a `linkset` for join performance. Linkset does not alter or infer severity/status.
|
||||
7. **Policy‑time only for effective findings**
|
||||
|
||||
* Only the Policy Engine can write `effective_finding_*` materializations.
|
||||
8. **Schema safety**
|
||||
|
||||
* Strict JSON schema validation at DB level; unknown fields reject writes.
|
||||
9. **Clock discipline**
|
||||
|
||||
* Timestamps are UTC, monotonic within a batch; collectors record `fetched_at` and `received_at`.
|
||||
|
||||
### 3.2 Data model
|
||||
|
||||
#### 3.2.1 `advisory_raw` (Mongo collection)
|
||||
|
||||
```json
|
||||
{
|
||||
"_id": "advisory_raw:osv:GHSA-xxxx-....:v3",
|
||||
"source": {
|
||||
"vendor": "OSV",
|
||||
"stream": "github",
|
||||
"api": "https://api.osv.dev/v1/.../GHSA-...",
|
||||
"collector_version": "conseiller/1.7.3"
|
||||
},
|
||||
"upstream": {
|
||||
"upstream_id": "GHSA-xxxx-....",
|
||||
"document_version": "2024-09-01T12:13:14Z",
|
||||
"fetched_at": "2025-01-02T03:04:05Z",
|
||||
"received_at": "2025-01-02T03:04:06Z",
|
||||
"content_hash": "sha256:...",
|
||||
"signature": {
|
||||
"present": true,
|
||||
"format": "dsse",
|
||||
"key_id": "rekor:.../key/abc",
|
||||
"sig": "base64..."
|
||||
}
|
||||
},
|
||||
"content": {
|
||||
"format": "OSV",
|
||||
"spec_version": "1.6",
|
||||
"raw": { /* full upstream JSON, unmodified */ }
|
||||
},
|
||||
"identifiers": {
|
||||
"cve": ["CVE-2023-1234"],
|
||||
"ghsa": ["GHSA-xxxx-...."],
|
||||
"aliases": ["CVE-2023-1234", "GHSA-xxxx-...."]
|
||||
},
|
||||
"linkset": {
|
||||
"purls": ["pkg:npm/lodash@4.17.21", "pkg:maven/..."],
|
||||
"cpes": ["cpe:2.3:a:..."],
|
||||
"references": [
|
||||
{"type":"advisory","url":"https://..."},
|
||||
{"type":"fix","url":"https://..."}
|
||||
],
|
||||
"reconciled_from": ["content.raw.affected.ranges", "content.raw.pkg"]
|
||||
},
|
||||
"supersedes": "advisory_raw:osv:GHSA-xxxx-....:v2",
|
||||
"tenant": "default"
|
||||
}
|
||||
```
|
||||
|
||||
> Note: No `severity`, no `cvss`, no `effective_*`. If the upstream payload includes CVSS, it stays inside `content.raw` and is not promoted or normalized at ingestion.
|
||||
|
||||
#### 3.2.2 `vex_raw` (Mongo collection)
|
||||
|
||||
```json
|
||||
{
|
||||
"_id": "vex_raw:vendorX:doc-123:v4",
|
||||
"source": {
|
||||
"vendor": "VendorX",
|
||||
"stream": "vex",
|
||||
"api": "https://.../vex/doc-123",
|
||||
"collector_version": "excitator/0.9.2"
|
||||
},
|
||||
"upstream": {
|
||||
"upstream_id": "doc-123",
|
||||
"document_version": "2025-01-15T08:09:10Z",
|
||||
"fetched_at": "2025-01-16T01:02:03Z",
|
||||
"received_at": "2025-01-16T01:02:03Z",
|
||||
"content_hash": "sha256:...",
|
||||
"signature": { "present": true, "format": "cms", "key_id": "kid:...", "sig": "..." }
|
||||
},
|
||||
"content": {
|
||||
"format": "CycloneDX-VEX", // or "CSAF-VEX"
|
||||
"spec_version": "1.5",
|
||||
"raw": { /* full upstream VEX */ }
|
||||
},
|
||||
"identifiers": {
|
||||
"statements": [
|
||||
{
|
||||
"advisory_ids": ["CVE-2023-1234","GHSA-..."],
|
||||
"component_purls": ["pkg:deb/openssl@1.1.1"],
|
||||
"status": "not_affected",
|
||||
"justification": "component_not_present"
|
||||
}
|
||||
]
|
||||
},
|
||||
"linkset": {
|
||||
"purls": ["pkg:deb/openssl@1.1.1"],
|
||||
"cves": ["CVE-2023-1234"],
|
||||
"ghsas": ["GHSA-..."]
|
||||
},
|
||||
"supersedes": "vex_raw:vendorX:doc-123:v3",
|
||||
"tenant": "default"
|
||||
}
|
||||
```
|
||||
|
||||
> VEX statuses remain as raw facts. No cross‑provider consensus is computed here.
|
||||
|
||||
### 3.3 Database validation
|
||||
|
||||
* MongoDB JSON Schema validators on both collections:
|
||||
|
||||
* Reject forbidden fields at the top level.
|
||||
* Enforce presence of `source`, `upstream`, `content`, `linkset`, `tenant`.
|
||||
* Enforce string formats for timestamps and hashes.
|
||||
|
||||
### 3.4 Write paths
|
||||
|
||||
1. **Collector fetches upstream**
|
||||
|
||||
* Normalize transport (gzip/json), compute `content_hash`, verify signature if available.
|
||||
2. **Build raw doc**
|
||||
|
||||
* Populate `source`, `upstream`, `content.raw`, `identifiers`, `linkset`.
|
||||
3. **Idempotent upsert**
|
||||
|
||||
* Lookup by `(source.vendor, upstream.upstream_id, upstream.content_hash)`. If exists, skip; if new content hash, insert new revision with `supersedes`.
|
||||
4. **AOC guard**
|
||||
|
||||
* Runtime interceptor inspects write payload; if any forbidden field detected, reject with `ERR_AOC_001`.
|
||||
5. **Metrics**
|
||||
|
||||
* Emit `ingestion_write_ok` or `ingestion_write_reject` with reason code.
|
||||
|
||||
### 3.5 Read paths (ingestion scope)
|
||||
|
||||
* Allow only listing, getting raw docs, and searching by linkset. No endpoints return “effective findings” from ingestion services.
|
||||
|
||||
### 3.6 Error codes
|
||||
|
||||
| Code | Meaning | HTTP |
|
||||
| ------------- | ------------------------------------------------------------ | ---- |
|
||||
| `ERR_AOC_001` | Forbidden field present (severity/consensus/normalized data) | 400 |
|
||||
| `ERR_AOC_002` | Merge attempt detected (multiple upstreams fused) | 400 |
|
||||
| `ERR_AOC_003` | Idempotency violation (duplicate without supersedes) | 409 |
|
||||
| `ERR_AOC_004` | Missing provenance fields | 422 |
|
||||
| `ERR_AOC_005` | Signature/checksum mismatch | 422 |
|
||||
| `ERR_AOC_006` | Attempt to write effective findings from ingestion context | 403 |
|
||||
| `ERR_AOC_007` | Unknown top‑level fields (schema violation) | 400 |
|
||||
|
||||
### 3.7 AOC Verifier
|
||||
|
||||
A build‑time and runtime safeguard:
|
||||
|
||||
* **Static checks (CI)**
|
||||
|
||||
* Block imports of `*.Policy*` or `*.Merge*` from ingestion modules.
|
||||
* AST lint rule: any write to `advisory_raw` or `vex_raw` setting a forbidden key fails the build.
|
||||
* **Runtime checks**
|
||||
|
||||
* Repository layer interceptor inspects documents before insert/update; rejects forbidden fields and multi‑source merges.
|
||||
* **Drift detection job**
|
||||
|
||||
* Nightly job scans newest N docs; if violation found, pages ops and blocks new pipeline runs.
|
||||
|
||||
> **Imposed rule:** Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.
|
||||
|
||||
### 3.8 Indexing strategy
|
||||
|
||||
* `advisory_raw`:
|
||||
|
||||
* `{ "identifiers.cve": 1 }`, `{ "identifiers.ghsa": 1 }`, `{ "linkset.purls": 1 }`, `{ "source.vendor": 1, "upstream.upstream_id": 1, "upstream.content_hash": 1 }` (unique), `{ "tenant": 1 }`.
|
||||
* `vex_raw`:
|
||||
|
||||
* `{ "identifiers.statements.advisory_ids": 1 }`, `{ "linkset.purls": 1 }`, `{ "source.vendor": 1, "upstream.upstream_id": 1, "upstream.content_hash": 1 }` (unique), `{ "tenant": 1 }`.
|
||||
|
||||
### 3.9 Interaction with Policy Engine
|
||||
|
||||
* Policy Engine pulls raw docs by identifiers/linksets and computes:
|
||||
|
||||
* De‑dup/merge per policy
|
||||
* Consensus for VEX statements
|
||||
* Severity normalization and risk scoring
|
||||
* Writes **only** to `effective_finding_{policyId}` collections.
|
||||
|
||||
A dedicated write guard refuses `effective_finding_*` writes from any caller that isn’t the Policy Engine service identity.
|
||||
|
||||
> **Imposed rule:** Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.
|
||||
|
||||
### 3.10 Migration plan
|
||||
|
||||
1. **Freeze ingestion writes** except raw pass‑through.
|
||||
2. **Backfill**: copy existing ingestion collections to `_backup_*`.
|
||||
3. **Strip forbidden fields** from raw copies, move them into a temporary `advisory_view_legacy` used only by Policy Engine for parity.
|
||||
4. **Enable DB schema validators**.
|
||||
5. **Run collectors** in dry‑run; ensure only allowed keys land.
|
||||
6. **Switch Policy Engine** to pull exclusively from `*_raw` and to compute everything else.
|
||||
7. **Delete legacy normalized fields** in ingestion codepaths.
|
||||
8. **Enable runtime guards** and CI lint.
|
||||
|
||||
### 3.11 Observability
|
||||
|
||||
* Metrics:
|
||||
|
||||
* `aoc_violation_total{code=...}`, `ingestion_write_total{result=ok|reject}`, `ingestion_signature_verified_total{result=ok|fail}`, `ingestion_latency_seconds`, `advisory_revision_count`.
|
||||
* Tracing: span `ingest.fetch`, `ingest.transform`, `ingest.write`, `aoc.guard`.
|
||||
* Logs: include `tenant`, `source.vendor`, `upstream.upstream_id`, `content_hash`, `correlation_id`.
|
||||
|
||||
> **Imposed rule:** Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.
|
||||
|
||||
### 3.12 Security and tenancy
|
||||
|
||||
* Every raw doc carries a `tenant` field.
|
||||
* Authority enforces `advisory:ingest` and `vex:ingest` scopes for ingestion endpoints.
|
||||
* Cross‑tenant reads/writes are blocked by default.
|
||||
* Secrets never logged; signatures verified with pinned trust stores.
|
||||
|
||||
> **Imposed rule:** Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.
|
||||
|
||||
### 3.13 CLI and Console behavior
|
||||
|
||||
* **CLI**
|
||||
|
||||
* `stella sources ingest --dry-run` prints would‑write payload and explicitly shows that no severity/status fields are present.
|
||||
* `stella aoc verify` scans last K documents and reports violations with exit codes.
|
||||
* **Console**
|
||||
|
||||
* Sources dashboard shows AOC pass/fail per job, most recent violation codes, and a drill‑down to the offending document.
|
||||
|
||||
> **Imposed rule:** Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.
|
||||
|
||||
---
|
||||
|
||||
## 4) API surface (ingestion scope)
|
||||
|
||||
### 4.1 Conseiller (Advisories)
|
||||
|
||||
* `POST /ingest/advisory`
|
||||
|
||||
* Body: raw upstream advisory with metadata; server constructs document, not the client.
|
||||
* Rejections: `ERR_AOC_00x` per table above.
|
||||
* `GET /advisories/raw/{id}`
|
||||
* `GET /advisories/raw?cve=CVE-...&purl=pkg:...&tenant=...`
|
||||
* `GET /advisories/raw/{id}/provenance`
|
||||
* `POST /aoc/verify?since=ISO8601` returns summary stats and first N violations.
|
||||
|
||||
### 4.2 Excitator (VEX)
|
||||
|
||||
* `POST /ingest/vex`
|
||||
* `GET /vex/raw/{id}`
|
||||
* `GET /vex/raw?advisory_id=CVE-...&purl=pkg:...`
|
||||
* `POST /aoc/verify?since=ISO8601`
|
||||
|
||||
All endpoints require `tenant` scope and appropriate `:write` or `:read`.
|
||||
|
||||
> **Imposed rule:** Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.
|
||||
|
||||
---
|
||||
|
||||
## 5) Example: end‑to‑end flow
|
||||
|
||||
1. Collector fetches `GHSA-1234` from OSV.
|
||||
2. Build `advisory_raw` with linkset PURLs.
|
||||
3. Insert; AOC guard approves.
|
||||
4. Policy Engine later evaluates SBOM `S-42` under `policy P-7`, reads raw advisory and any VEX raw docs, computes effective findings, and writes to `effective_finding_P-7`.
|
||||
5. CLI `stella aoc verify --since 24h` returns `0` violations.
|
||||
|
||||
---
|
||||
|
||||
## 6) Implementation tasks
|
||||
|
||||
Breakdown by component with exact work items. Each section ends with the imposed sentence you requested.
|
||||
|
||||
### 6.1 Conseiller (advisory ingestion, WS + Worker)
|
||||
|
||||
* [ ] Add Mongo JSON schema validation for `advisory_raw`.
|
||||
* [ ] Implement repository layer with **write interceptors** that reject forbidden fields.
|
||||
* [ ] Compute `linkset` from upstream using deterministic mappers.
|
||||
* [ ] Enforce idempotency by unique index on `(source.vendor, upstream.upstream_id, upstream.content_hash, tenant)`.
|
||||
* [ ] Remove any normalization pipelines; relocate to Policy Engine.
|
||||
* [ ] Add `POST /ingest/advisory` and `GET /advisories/raw*` endpoints with Authority scope checks.
|
||||
* [ ] Emit observability metrics and traces.
|
||||
* [ ] Unit tests: schema violations, idempotency, supersedes chain, forbidden fields.
|
||||
* [ ] Integration tests: large batch ingest, linkset correctness against golden fixtures.
|
||||
|
||||
**Imposed rule:** Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.
|
||||
|
||||
### 6.2 Excitator (VEX ingestion, WS + Worker)
|
||||
|
||||
* [ ] Add Mongo JSON schema validation for `vex_raw`.
|
||||
* [ ] Implement repository layer guard identical to Conseiller.
|
||||
* [ ] Deterministic `linkset` extraction for advisory IDs and PURLs.
|
||||
* [ ] Endpoints `POST /ingest/vex`, `GET /vex/raw*` with scopes.
|
||||
* [ ] Remove any consensus or folding logic; leave VEX statements as raw.
|
||||
* [ ] Tests as per Conseiller, with rich fixtures for CycloneDX‑VEX and CSAF.
|
||||
|
||||
**Imposed rule:** Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.
|
||||
|
||||
### 6.3 Web API shared library
|
||||
|
||||
* [ ] Define `AOCForbiddenKeys` and export for both services.
|
||||
* [ ] Provide `AOCWriteGuard` middleware and `AOCError` types.
|
||||
* [ ] Provide `ProvenanceBuilder` utility.
|
||||
* [ ] Provide `SignatureVerifier` and `Checksum` helpers.
|
||||
|
||||
**Imposed rule:** Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.
|
||||
|
||||
### 6.4 Policy Engine
|
||||
|
||||
* [ ] Block any import/use from ingestion modules by lint rule.
|
||||
* [ ] Add hard gate on `effective_finding_*` writes that verifies caller identity is Policy Engine.
|
||||
* [ ] Update readers to pull fields only from `content.raw`, `identifiers`, `linkset`, not any legacy normalized fields.
|
||||
|
||||
**Imposed rule:** Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.
|
||||
|
||||
### 6.5 Authority
|
||||
|
||||
* [ ] Introduce scopes: `advisory:ingest`, `advisory:read`, `vex:ingest`, `vex:read`, `aoc:verify`.
|
||||
* [ ] Add `tenant` claim propagation to ingestion services.
|
||||
|
||||
**Imposed rule:** Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.
|
||||
|
||||
### 6.6 CLI
|
||||
|
||||
* [ ] `stella sources ingest --dry-run` and `stella aoc verify` commands.
|
||||
* [ ] Exit codes mapping to `ERR_AOC_00x`.
|
||||
* [ ] JSON output schema including violation list.
|
||||
|
||||
**Imposed rule:** Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.
|
||||
|
||||
### 6.7 Console
|
||||
|
||||
* [ ] Sources dashboard tiles: last run, AOC violations, top error codes.
|
||||
* [ ] Drill‑down page rendering offending doc with highlight on forbidden keys.
|
||||
* [ ] “Verify last 24h” action calling the AOC Verifier endpoint.
|
||||
|
||||
**Imposed rule:** Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.
|
||||
|
||||
### 6.8 CI/CD
|
||||
|
||||
* [ ] AST linter to forbid writes of banned keys in ingestion modules.
|
||||
* [ ] Unit test coverage gates for AOC guard code.
|
||||
* [ ] Pipeline stage that runs `stella aoc verify` against seeded DB snapshots.
|
||||
|
||||
**Imposed rule:** Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.
|
||||
|
||||
---
|
||||
|
||||
## 7) Documentation changes (create/update these files)
|
||||
|
||||
1. **`/docs/ingestion/aggregation-only-contract.md`**
|
||||
|
||||
* Add: philosophy, invariants, schemas for `advisory_raw`/`vex_raw`, error codes, linkset definition, examples, idempotency rules, supersedes, API references, migration steps, observability, security.
|
||||
2. **`/docs/architecture/overview.md`**
|
||||
|
||||
* Update system diagram to show AOC boundary and raw stores; add sequence diagram: fetch → guard → raw insert → policy evaluation.
|
||||
3. **`/docs/architecture/policy-engine.md`**
|
||||
|
||||
* Clarify ingestion boundary; list inputs consumed from raw; note that any severity/consensus is policy‑time only.
|
||||
4. **`/docs/ui/console.md`**
|
||||
|
||||
* Add Sources dashboard section: AOC tiles and violation drill‑down.
|
||||
5. **`/docs/cli/cli-reference.md`**
|
||||
|
||||
* Add `stella aoc verify` and `stella sources ingest --dry-run` usage and exit codes.
|
||||
6. **`/docs/observability/observability.md`**
|
||||
|
||||
* Document new metrics, traces, logs keys for AOC.
|
||||
7. **`/docs/security/authority-scopes.md`**
|
||||
|
||||
* Add new scopes and tenancy enforcement for ingestion endpoints.
|
||||
8. **`/docs/deploy/containers.md`**
|
||||
|
||||
* Note DB validators must be enabled; environment flags for AOC guards; read‑only user for verify endpoint.
|
||||
|
||||
Each file should include a “Compliance checklist” subsection for AOC.
|
||||
|
||||
> **Imposed rule:** Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.
|
||||
|
||||
---
|
||||
|
||||
## 8) Acceptance criteria
|
||||
|
||||
* DB validators are active and reject writes with forbidden fields.
|
||||
* AOC runtime guards log and reject violations with correct error codes.
|
||||
* CI linter prevents shipping code that writes forbidden keys to raw stores.
|
||||
* Ingestion of known fixture sets results in zero normalized fields outside `content.raw`.
|
||||
* Policy Engine is the only writer of `effective_finding_*` materializations.
|
||||
* CLI `stella aoc verify` returns success on clean datasets and non‑zero on seeded violations.
|
||||
* Console shows AOC status and violation drill‑downs.
|
||||
|
||||
---
|
||||
|
||||
## 9) Risks and mitigations
|
||||
|
||||
* **Collector drift**: new upstream fields tempt developers to normalize.
|
||||
|
||||
* Mitigation: CI linter + guard + schema validators; require RFC to extend linkset.
|
||||
* **Performance impact**: extra validation on write.
|
||||
|
||||
* Mitigation: guard is O(number of keys) and schema check is bounded; indexes sized appropriately.
|
||||
* **Migration complexity**: moving legacy normalized fields out.
|
||||
|
||||
* Mitigation: temporary `advisory_view_legacy` for parity; stepwise cutover.
|
||||
* **Tenant leakage**: missing tenant on write.
|
||||
|
||||
* Mitigation: schema requires `tenant`; middleware injects and asserts.
|
||||
|
||||
---
|
||||
|
||||
## 10) Test plan
|
||||
|
||||
* **Unit tests**
|
||||
|
||||
* Guard rejects forbidden keys; idempotency; supersedes chain; provenance required.
|
||||
* Signature verification paths: good, bad, absent.
|
||||
* **Property tests**
|
||||
|
||||
* Randomized upstream docs never produce forbidden keys at top level.
|
||||
* **Integration tests**
|
||||
|
||||
* Batch ingest of 50k advisories: throughput, zero violations.
|
||||
* Mixed VEX sources with contradictory statements remain separate in raw.
|
||||
* **Contract tests**
|
||||
|
||||
* Policy Engine refuses to run without raw inputs; writes only to `effective_finding_*`.
|
||||
* **End‑to‑end**
|
||||
|
||||
* Seed SBOM + advisories + VEX; ensure findings are identical pre/post migration.
|
||||
|
||||
---
|
||||
|
||||
## 11) Developer checklists
|
||||
|
||||
**Definition of Ready**
|
||||
|
||||
* Upstream spec reference attached.
|
||||
* Linkset mappers defined.
|
||||
* Example fixtures added.
|
||||
|
||||
**Definition of Done**
|
||||
|
||||
* DB validators deployed and tested.
|
||||
* Runtime guards enabled.
|
||||
* CI linter merged and enforced.
|
||||
* Docs updated (files in section 7).
|
||||
* Metrics visible on dashboard.
|
||||
* CLI verify passes.
|
||||
|
||||
---
|
||||
|
||||
## 12) Glossary
|
||||
|
||||
* **Raw document**: exact upstream content plus provenance, with join hints.
|
||||
* **Linkset**: PURLs/CPEs/IDs extracted to accelerate joins later.
|
||||
* **Supersedes**: pointer from a newer raw doc to the previous revision of the same upstream doc.
|
||||
* **Policy‑time**: evaluation phase where merges, consensus, and severity are computed.
|
||||
* **AOC**: Aggregation‑Only Contract.
|
||||
|
||||
---
|
||||
|
||||
### Final imposed reminder
|
||||
|
||||
**Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.**
|
||||
566
docs/implplan/EPIC_10.md
Normal file
566
docs/implplan/EPIC_10.md
Normal file
@@ -0,0 +1,566 @@
|
||||
Fine. Here’s your next brick of “maximum documentation.” Try not to drop it on your foot.
|
||||
|
||||
> **Imposed rule:** Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.
|
||||
|
||||
---
|
||||
|
||||
# Epic 10: Export Center (JSON, Trivy DB, Mirror bundles)
|
||||
|
||||
**Short name:** `Export Center`
|
||||
**Primary service:** `exporter`
|
||||
**Surfaces:** Console (Web UI), CLI, Web API
|
||||
**Touches:** Conseiller (Feedser), Excitator (Vexer), SBOM Service, Policy Engine, VEX Consensus Lens, Findings Ledger, Authority (authN/Z), Object Storage, Orchestrator, Signing/Attestation, Telemetry
|
||||
|
||||
**AOC ground rule:** Conseiller and Excitator aggregate but never merge. The Export Center serializes evidence and policy results; it does not rewrite or “improve” your data in-flight.
|
||||
|
||||
---
|
||||
|
||||
## 1) What it is
|
||||
|
||||
The Export Center is the unified system for packaging StellaOps data into portable, verifiable bundles:
|
||||
|
||||
* **Canonical JSON exports** for advisories, VEX, SBOMs, findings, and policy-evaluation snapshots.
|
||||
* **Trivy DB compatible bundles** so downstream scanners can use Stella’s curated vulnerability knowledge without custom integrations.
|
||||
* **Mirror bundles** for air‑gapped or disconnected environments containing raw evidence, normalized records, indexes, and policy snapshots, with provenance, signatures, and optional encryption.
|
||||
|
||||
It centralizes format adapters, compliance and provenance, scheduling, versioning, and distribution (download, OCI push, file share). Every export is reproducible by `run_id`, cryptographically signed, and audit‑traceable back to source artifacts.
|
||||
|
||||
---
|
||||
|
||||
## 2) Why (brief)
|
||||
|
||||
Teams need to move results into other scanners, CI systems, and isolated networks without babysitting ten different scripts. The Export Center gives a single, policy‑aware, verifiable exit point that doesn’t surprise compliance or set your ops team on fire.
|
||||
|
||||
---
|
||||
|
||||
## 3) How it should work (maximum detail)
|
||||
|
||||
### 3.1 Capabilities
|
||||
|
||||
* **Profiles**
|
||||
|
||||
* `json:raw` canonical JSON lines for advisories, VEX, SBOMs.
|
||||
* `json:policy` adds policy evaluation results (allow/deny/risk, rationales).
|
||||
* `trivy:db` Trivy DB‑compatible export (core vulnerability DB).
|
||||
* `trivy:java-db` optional Java ecosystem DB export if enabled.
|
||||
* `mirror:full` air‑gap bundle with raw + normalized + indexes + policy + VEX consensus + SBOMs.
|
||||
* `mirror:delta` incremental bundle based on a previous export manifest.
|
||||
|
||||
* **Scope selectors**
|
||||
|
||||
* By time window, product, ecosystem, package, image digest, repository, tenant, tag.
|
||||
* Include/exclude SBOMs, advisories, VEX, findings, policy snapshots.
|
||||
|
||||
* **Policy awareness**
|
||||
|
||||
* Optionally bake a **policy snapshot** into the bundle, including the policy version, inputs and decision traces.
|
||||
* Can export **raw** evidence only (AOC) or **raw + evaluated**.
|
||||
|
||||
* **Provenance & signing**
|
||||
|
||||
* Generate attestation metadata with source URIs, artifact hashes, schema versions, export profile and filters.
|
||||
* Sign manifests and bundle using configured KMS; support detached and in‑bundle signatures.
|
||||
|
||||
* **Distribution**
|
||||
|
||||
* Download via Console or API (streaming).
|
||||
* Push to OCI registry as an artifact/image with annotations.
|
||||
* Write to object storage prefix for batch pickup.
|
||||
|
||||
* **Scheduling & automation**
|
||||
|
||||
* One‑off, cron, and event‑triggered exports (e.g., after a “VEX consensus recompute” run).
|
||||
* Retention policies and automatic expiry for old bundles.
|
||||
|
||||
* **Observability**
|
||||
|
||||
* Export metrics, throughput, size, downstream pull counts (when pushed to registry with report backs).
|
||||
|
||||
### 3.2 Architecture
|
||||
|
||||
* **exporter (service)**
|
||||
|
||||
* Orchestrates export runs, gathers records from the ledger and indexes, calls format adapters, writes bundles, signs, and publishes distribution tasks.
|
||||
* Stateless workers pull “export.jobs” from the orchestrator, stream data, and write manifests into object storage.
|
||||
|
||||
* **format adapters**
|
||||
|
||||
* Pluggable adapters:
|
||||
|
||||
* `adapter-json`: canonicalized JSONL writers per record type.
|
||||
* `adapter-trivy`: translates Stella’s normalized advisory model into Trivy DB format (and Java DB if enabled).
|
||||
* `adapter-mirror`: constructs a portable filesystem/OCI layout with manifests, indexes, and data subtrees.
|
||||
|
||||
* **manifesting & provenance**
|
||||
|
||||
* `export.json` (export manifest): profile, filters, counts, schema versions, content checksums, start/finish times, inputs list (artifact ids + hashes).
|
||||
* `provenance.json`: full chain back to source runs and artifacts; linked signatures.
|
||||
|
||||
* **distribution engines**
|
||||
|
||||
* `dist-http` streaming for downloads.
|
||||
* `dist-oci` layer writer with descriptors and annotations.
|
||||
* `dist-objstore` for staging to buckets.
|
||||
|
||||
* **security**
|
||||
|
||||
* Tenant scoping, RBAC on export creation and retrieval.
|
||||
* Optional in‑bundle encryption (age/AES‑GCM) with key wrapping using KMS.
|
||||
|
||||
### 3.3 Data model (selected tables)
|
||||
|
||||
* `export_profiles`
|
||||
|
||||
* `id`, `name`, `kind` (`json|trivy|mirror`), `variant` (`raw|policy|db|java-db|full|delta`), `config_json`, `created_by`, `created_at`.
|
||||
|
||||
* `export_runs`
|
||||
|
||||
* `id`, `profile_id`, `trigger` (`manual|schedule|event|api`), `state`, `filters_json`, `policy_version`, `started_at`, `finished_at`, `artifact_uri`, `size_bytes`, `sig_uri`, `provenance_uri`, `tenant_id`, `error_class`, `error_message`.
|
||||
|
||||
* `export_inputs`
|
||||
|
||||
* Link table between `export_runs` and source artifacts. `export_run_id`, `artifact_id`, `hash`.
|
||||
|
||||
* `export_distributions`
|
||||
|
||||
* `export_run_id`, `type` (`download|oci|objstore`), `target`, `state`, `meta_json`, `created_at`, `updated_at`.
|
||||
|
||||
### 3.4 Canonical file layouts
|
||||
|
||||
**JSON profile output**
|
||||
Directory layout under export root:
|
||||
|
||||
```
|
||||
/export/
|
||||
export.json # export manifest
|
||||
provenance.json # provenance and source artifact chain
|
||||
signatures/
|
||||
export.sig # detached signature for export.json
|
||||
advisories/
|
||||
normalized.jsonl # normalized advisory records
|
||||
vex/
|
||||
normalized.jsonl # normalized VEX records
|
||||
sboms/
|
||||
<subject>/sbom.spdx.json # one per subject; SPDX JSON or CycloneDX JSON
|
||||
findings/
|
||||
policy_evaluated.jsonl # if profile=json:policy
|
||||
```
|
||||
|
||||
**Trivy DB profile output**
|
||||
Produced as a compressed artifact:
|
||||
|
||||
```
|
||||
/export/
|
||||
export.json
|
||||
provenance.json
|
||||
trivy/
|
||||
db.bundle # Trivy DB compatible archive
|
||||
java-db.bundle # optional Java DB bundle (if enabled)
|
||||
signatures/
|
||||
trivy-db.sig
|
||||
```
|
||||
|
||||
Notes:
|
||||
|
||||
* The adapter keeps an internal mapping of Stella normalized fields to Trivy’s expected fields and namespaces. The mapping is versioned to track upstream schema evolution.
|
||||
|
||||
**Mirror bundle (filesystem layout)**
|
||||
|
||||
```
|
||||
/mirror/
|
||||
manifest.yaml # high-level bundle manifest (profile, filters, counts)
|
||||
export.json # same as JSON profile
|
||||
provenance.json
|
||||
indexes/
|
||||
advisories.index.json # quick lookups (pkg -> advisory ids)
|
||||
vex.index.json
|
||||
sbom.index.json
|
||||
advisories/raw/...
|
||||
advisories/normalized/...
|
||||
vex/raw/...
|
||||
vex/normalized/...
|
||||
sboms/raw/...
|
||||
sboms/graph/...
|
||||
policy/
|
||||
snapshot.yaml # full policy set used for evaluation
|
||||
evaluations.jsonl # decision outputs if requested
|
||||
consensus/
|
||||
vex_consensus.jsonl
|
||||
signatures/
|
||||
manifest.sig
|
||||
export.sig
|
||||
README.md
|
||||
```
|
||||
|
||||
**Mirror bundle (OCI layout)**
|
||||
Following standard OCI image artifact layout with annotations (`org.opencontainers.artifact.description`, `com.stella.export.profile`, `com.stella.export.filters`), and manifest lists for large bundles.
|
||||
|
||||
### 3.5 Export workflow
|
||||
|
||||
1. **Plan**
|
||||
Exporter computes candidates based on filters. For `mirror:delta`, compares with previous manifest to compute changes.
|
||||
|
||||
2. **Stream & write**
|
||||
Records are streamed from the Findings Ledger and stores. Writers are forward‑only, emitting JSONL or adapter‑specific structures, chunked for memory safety.
|
||||
|
||||
3. **Sign & attest**
|
||||
Once all content hashes are stable, Export Center writes `export.json`, `provenance.json`, and signs using KMS. Optional encryption wraps data layers.
|
||||
|
||||
4. **Distribute**
|
||||
Depending on profile settings, it exposes a download URL, pushes an OCI artifact, or writes to object storage. Distribution metadata is recorded.
|
||||
|
||||
5. **Audit & retention**
|
||||
Run, manifest, and signatures are immutable. Retention policy prunes large data folders after N days with manifests retained longer.
|
||||
|
||||
### 3.6 APIs
|
||||
|
||||
```
|
||||
POST /export/profiles
|
||||
GET /export/profiles?kind=&variant=
|
||||
GET /export/profiles/{id}
|
||||
PATCH /export/profiles/{id}
|
||||
DELETE /export/profiles/{id}
|
||||
|
||||
POST /export/runs
|
||||
GET /export/runs?state=&profile_id=&from=&to=&tenant_id=
|
||||
GET /export/runs/{run_id}
|
||||
POST /export/runs/{run_id}/cancel
|
||||
|
||||
GET /export/runs/{run_id}/download # presigned URL or streaming
|
||||
POST /export/runs/{run_id}/distribute # { "type":"oci|objstore", "target":"..." }
|
||||
GET /export/runs/{run_id}/manifest # export.json
|
||||
GET /export/runs/{run_id}/provenance
|
||||
GET /export/runs/{run_id}/signatures
|
||||
|
||||
GET /export/metrics/overview
|
||||
WS /export/streams/updates
|
||||
```
|
||||
|
||||
**Request example (create run):**
|
||||
|
||||
```json
|
||||
{
|
||||
"profile_id": "prof_json_policy",
|
||||
"filters": {
|
||||
"time_from": "2025-01-01T00:00:00Z",
|
||||
"time_to": "2025-01-31T23:59:59Z",
|
||||
"ecosystems": ["pypi", "npm"],
|
||||
"include": ["advisories", "vex", "sboms", "findings"]
|
||||
},
|
||||
"distribution": { "type": "download" },
|
||||
"policy_version": "pol-v1.8.2"
|
||||
}
|
||||
```
|
||||
|
||||
### 3.7 CLI
|
||||
|
||||
```
|
||||
stella export profiles list --kind mirror
|
||||
stella export profiles create --file profile.yaml
|
||||
stella export run create --profile prof_json_policy --from 2025-01-01 --to 2025-01-31 --include advisories,vex,sboms --download
|
||||
stella export run status <run-id>
|
||||
stella export run cancel <run-id>
|
||||
stella export run get <run-id> --manifest
|
||||
stella export run download <run-id> --out export-jan.tar.zst
|
||||
stella export distribute <run-id> --oci ghcr.io/org/stella-export:jan2025
|
||||
stella export verify <bundle> --manifest export.json --sig signatures/export.sig
|
||||
```
|
||||
|
||||
Exit codes: `0` ok, `2` bad args, `4` not found, `5` denied, `6` integrity failed, `8` export error.
|
||||
|
||||
### 3.8 RBAC & security
|
||||
|
||||
* Roles:
|
||||
|
||||
* `Export.Viewer`: list runs, download completed bundles.
|
||||
* `Export.Operator`: create runs, cancel, schedule, distribute.
|
||||
* `Export.Admin`: manage profiles, set retention, manage signing keys.
|
||||
|
||||
* Tenancy:
|
||||
|
||||
* Every run and artifact scoped by tenant; cross‑tenant export is disallowed.
|
||||
|
||||
* Secrets:
|
||||
|
||||
* KMS references for signing and encryption; never store private keys.
|
||||
|
||||
* PII & redaction:
|
||||
|
||||
* Exporters must not include secrets or credentials. Redaction rules enforced at writer level with schema‑based allowlists.
|
||||
|
||||
### 3.9 Observability
|
||||
|
||||
* Metrics:
|
||||
|
||||
* `export_bytes_total{profile,tenant}`
|
||||
* `export_records_total{type}`
|
||||
* `export_duration_ms{profile}`
|
||||
* `export_failures_total{error_class}`
|
||||
* `export_distributions_total{type}`
|
||||
* `export_verify_fail_total`
|
||||
|
||||
* Traces:
|
||||
|
||||
* Spans per export phase: plan, stream, write, sign, distribute; baggage includes `export_run_id`.
|
||||
|
||||
* Logs:
|
||||
|
||||
* Structured JSON with counts, sizes, hashes, and redaction hints.
|
||||
|
||||
### 3.10 Performance targets
|
||||
|
||||
* Stream throughput ≥ 25k records/sec per worker for JSONL writing with compression.
|
||||
* Trivy bundle generation for 1M advisories ≤ 8 minutes on a standard worker.
|
||||
* Mirror delta export for 5% change set ≤ 2 minutes.
|
||||
|
||||
### 3.11 Edge cases & behavior
|
||||
|
||||
* **Schema drift**: adapter refuses to emit unknown fields without explicit mapping; run fails with `error_class=schema_mismatch`.
|
||||
* **Oversized bundles**: automatic sharding by time or content type; mirror OCI uses multi‑manifest indices.
|
||||
* **Missing policy snapshot**: profile `json:policy` will auto‑pull latest version unless pinned; pinning is recommended for reproducibility.
|
||||
* **Duplicate evidence**: writers dedupe by artifact hash and advisory id; AOC forbids merging.
|
||||
* **Air‑gap encryption**: if `encrypt=true`, mirror bundles require recipient public key material; decryption tooling documented.
|
||||
|
||||
---
|
||||
|
||||
## 4) Implementation plan
|
||||
|
||||
### 4.1 Modules
|
||||
|
||||
* **New service:** `src/ExportCenter/StellaOps.ExportCenter`
|
||||
|
||||
* `api/` REST + WS
|
||||
* `planner/` scope planning, delta computation, sampling
|
||||
* `adapters/`
|
||||
|
||||
* `json/` canonical writers
|
||||
* `trivy/` db builders and schema mapping
|
||||
* `mirror/` fs/OCI builders, sharding, delta logic
|
||||
* `signing/` KMS clients, attestations
|
||||
* `dist/` download streaming, OCI push, object storage writer
|
||||
* `state/` repositories, migrations
|
||||
* `metrics/`, `audit/`, `security/`
|
||||
|
||||
* **SDK/CLI**
|
||||
|
||||
* `src/Cli/StellaOps.Cli` subcommands with streaming download and verification.
|
||||
|
||||
* **Console**
|
||||
|
||||
* `console/apps/export-center/` pages:
|
||||
|
||||
* Overview, Profiles, Runs, Run Detail, Distributions, Settings.
|
||||
* Components: `ExportPlanPreview`, `ProfileEditor`, `RunDiff`, `VerifyPanel`.
|
||||
|
||||
* **Existing services updates**
|
||||
|
||||
* Findings Ledger: new paginated streaming endpoints for advisories/VEX/SBOM/findings by filters and snapshots.
|
||||
* Policy Engine: “policy snapshot” exportable endpoint.
|
||||
* VEX Lens: consensus snapshot endpoint.
|
||||
|
||||
### 4.2 Packaging & deployment
|
||||
|
||||
* Containers:
|
||||
|
||||
* `stella/exporter:<ver>`
|
||||
* `stella/exporter-worker:<ver>` (optional separated worker pool)
|
||||
* Helm:
|
||||
|
||||
* WS replicas, concurrent export limits, default compression (`zstd`), default retention, KMS settings, OCI creds secrets.
|
||||
* DB migrations:
|
||||
|
||||
* Create `export_*` tables with proper indices (tenant, time, state).
|
||||
|
||||
### 4.3 Rollout
|
||||
|
||||
* Phase 1: JSON (raw, policy) and Mirror (full) as download only.
|
||||
* Phase 2: Trivy DB adapters, OCI distribution.
|
||||
* Phase 3: Mirror deltas, encryption, verification tooling, scheduling.
|
||||
|
||||
---
|
||||
|
||||
## 5) Documentation changes
|
||||
|
||||
Create/update the following docs; each page must end with the imposed rule statement.
|
||||
|
||||
1. `/docs/export-center/overview.md`
|
||||
Purpose, profiles, supported targets, AOC alignment, security model.
|
||||
|
||||
2. `/docs/export-center/architecture.md`
|
||||
Service components, adapters, manifests, signing, distribution flows.
|
||||
|
||||
3. `/docs/export-center/profiles.md`
|
||||
Profile schemas, examples, versioning, compatibility notes.
|
||||
|
||||
4. `/docs/export-center/api.md`
|
||||
All endpoints with request/response examples and error codes.
|
||||
|
||||
5. `/docs/export-center/cli.md`
|
||||
Commands with examples, scripts for CI/CD, verification.
|
||||
|
||||
6. `/docs/export-center/mirror-bundles.md`
|
||||
Filesystem and OCI layouts, delta exports, encryption, air‑gap import guide.
|
||||
|
||||
7. `/docs/export-center/trivy-adapter.md`
|
||||
Field mapping, supported ecosystems, compatibility and test matrix.
|
||||
|
||||
8. `/docs/export-center/provenance-and-signing.md`
|
||||
Manifest format, attestation details, verification process.
|
||||
|
||||
9. `/docs/operations/export-runbook.md`
|
||||
Common failures, recovery, tuning, capacity planning.
|
||||
|
||||
10. `/docs/security/export-hardening.md`
|
||||
RBAC, tenant isolation, secret redaction, encryption keys.
|
||||
|
||||
> **Imposed rule:** Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.
|
||||
|
||||
---
|
||||
|
||||
## 6) Engineering tasks
|
||||
|
||||
### Backend: exporter
|
||||
|
||||
* [ ] Migrations for `export_profiles`, `export_runs`, `export_inputs`, `export_distributions`.
|
||||
* [ ] Planner to resolve filters to iterators over advisory/VEX/SBOM/findings datasets with pagination.
|
||||
* [ ] JSON adapter: canonical JSONL writers with schema normalization and redaction enforcement.
|
||||
* [ ] Policy snapshot embedder: pull policy version and evaluation outputs when requested.
|
||||
* [ ] Trivy adapter: implement schema mapping, writer, integrity validation, and compatibility version flag.
|
||||
* [ ] Mirror adapter: filesystem and OCI writer, sharding, manifest creation, delta computation.
|
||||
* [ ] Signing/attestation using KMS; detached and embedded options.
|
||||
* [ ] Distribution engines: download streaming, OCI push, object storage staging.
|
||||
* [ ] API layer with async export run handling and WebSocket updates.
|
||||
* [ ] Rate limit and concurrency controls per tenant/profile.
|
||||
* [ ] Audit logging for all create/cancel/distribute/verify actions.
|
||||
|
||||
### Integrations
|
||||
|
||||
* [ ] Findings Ledger streaming APIs for each content type.
|
||||
* [ ] Policy Engine endpoint to return deterministic policy snapshot and decision set by run.
|
||||
* [ ] VEX Lens endpoint to expose consensus snapshot.
|
||||
|
||||
### Console
|
||||
|
||||
* [ ] Profiles CRUD with validation and test preview.
|
||||
* [ ] Create Run wizard with live count estimates and storage footprint prediction.
|
||||
* [ ] Runs list + detail page with manifest, provenance, and quick verify.
|
||||
* [ ] Download and distribution actions with progress and logs.
|
||||
* [ ] Verification panel to check signatures and hashes client‑side.
|
||||
|
||||
### CLI
|
||||
|
||||
* [ ] `stella export` commands as defined; include `verify` that checks signatures and hashes.
|
||||
* [ ] Auto‑resume of interrupted downloads with range requests.
|
||||
* [ ] Friendly error messages for schema mismatch and verification failure.
|
||||
|
||||
### Observability
|
||||
|
||||
* [ ] Metrics and traces per §3.9; dashboards for throughput, durations, failures, sizes.
|
||||
* [ ] Alerts for export failure rate and verify failures.
|
||||
|
||||
### Security & RBAC
|
||||
|
||||
* [ ] Enforce tenant scoping at query level; fuzz tests for leakage.
|
||||
* [ ] Role matrix checks on each API; Console hides forbidden actions.
|
||||
* [ ] Encryption test vectors and key rotation procedure.
|
||||
|
||||
### Docs
|
||||
|
||||
* [ ] Author all files in §5 with concrete examples and diagrams.
|
||||
* [ ] Cross‑link from Orchestrator, Policy Studio, VEX Lens, and SBOM docs to the Export Center pages.
|
||||
* [ ] Append imposed rule line at the end of each page.
|
||||
|
||||
> **Imposed rule:** Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.
|
||||
|
||||
---
|
||||
|
||||
## 7) Implementation notes per profile
|
||||
|
||||
### 7.1 JSON: raw
|
||||
|
||||
* **Content:** advisories.normalized, vex.normalized, sboms (SPDX/CycloneDX), optional findings.raw.
|
||||
* **Normalization:** enforce Stella field casing, timestamps in RFC3339, unicode NFC.
|
||||
* **Compression:** `.jsonl.zst` per file to allow split/merge.
|
||||
|
||||
### 7.2 JSON: policy
|
||||
|
||||
* **Adds:** `policy_snapshot` and `findings.policy_evaluated.jsonl` with decision, rule_id, rationale, inputs fingerprint.
|
||||
* **Determinism:** include `policy_version` and `inputs_hash`; replays should match exactly.
|
||||
|
||||
### 7.3 Trivy DB
|
||||
|
||||
* **Mapping:**
|
||||
|
||||
* Package name, ecosystem, version ranges, CVE/CWE/aliases, severity mapping, vendor statements, fixed versions.
|
||||
* Ensure namespace mapping avoids collisions (e.g., distro vs ecosystem).
|
||||
* **Compatibility:** version flag in manifest; adapter throws if upstream schema version not supported.
|
||||
* **Validation:** run post‑build sanity checks (counts, required indexes).
|
||||
|
||||
### 7.4 Mirror: full/delta
|
||||
|
||||
* **Full:** everything needed to spin up an isolated read‑only Stella mirror with local search.
|
||||
* **Delta:** compute changed/added/removed advisory ids, VEX statements, SBOM subjects; update indexes and manifest with `base_export_id`.
|
||||
* **Encryption:** if enabled, encrypt data subtrees; leave `manifest.yaml` unencrypted for discoverability unless `strict=true`.
|
||||
|
||||
---
|
||||
|
||||
## 8) Acceptance criteria
|
||||
|
||||
* Operators can create an export with filters, download it, verify signatures, and trace back to source artifacts via provenance.
|
||||
* Trivy adapter produces a bundle consumable by Trivy without custom flags (basic validation in CI).
|
||||
* Mirror bundle imports successfully in an air‑gapped “mirror‑reader” sample app and serves queries from indexes.
|
||||
* Policy‑aware exports include deterministic decisions matching the specified `policy_version`.
|
||||
* RBAC prevents a Viewer from creating or canceling exports; tenancy prevents cross‑tenant leakage.
|
||||
* Metrics and dashboards show per‑profile throughput and error classes; alerts trigger on failure spikes.
|
||||
* Export retries are idempotent and do not duplicate content; hashes stable across re‑runs with identical inputs.
|
||||
|
||||
---
|
||||
|
||||
## 9) Risks & mitigations
|
||||
|
||||
* **Upstream schema changes break Trivy export.**
|
||||
Mitigation: versioned adapter with compatibility gate; integration tests against known fixtures; fail early with clear remediation.
|
||||
|
||||
* **Bundle size explosion.**
|
||||
Mitigation: zstd compression, sharding, delta exports, content‑addressed storage reuse for mirror OCI.
|
||||
|
||||
* **Data leakage via exports.**
|
||||
Mitigation: strict allowlist schemas, redaction filters, RBAC, tenant scoping tests, encryption for mirror.
|
||||
|
||||
* **Non‑deterministic policy outputs.**
|
||||
Mitigation: pin policy version and inputs hash; snapshot embedded rules; deterministic evaluation mode only.
|
||||
|
||||
* **Slow downloads/timeouts.**
|
||||
Mitigation: streaming with range support, resumable downloads, CDN integration if needed.
|
||||
|
||||
---
|
||||
|
||||
## 10) Test plan
|
||||
|
||||
* **Unit**
|
||||
Schema normalization; Trivy mapping; mirror delta computation; manifest hashing; signing.
|
||||
|
||||
* **Integration**
|
||||
End‑to‑end export with each profile; verify bundles; replay determinism of policy exports; OCI push/pull.
|
||||
|
||||
* **Compatibility**
|
||||
Validate Trivy bundle against a matrix of versions; import mirror bundle into a reference reader and run queries.
|
||||
|
||||
* **Security**
|
||||
Tenant isolation fuzzing; redaction checks; encryption round‑trip; signature verification with rotated keys.
|
||||
|
||||
* **Performance**
|
||||
Large dataset generation; parallel writer stress; OCI multi‑manifest publishing; download resume under packet loss.
|
||||
|
||||
* **Chaos**
|
||||
Kill exporter mid‑write; ensure resume or clean failure without partial corrupt bundles.
|
||||
|
||||
---
|
||||
|
||||
## 11) Philosophy
|
||||
|
||||
* **Ports, not prisons.** Exports should free your data to move with integrity and context, not trap it in a proprietary maze.
|
||||
* **Reproducible or it didn’t happen.** Every bit derived from known inputs, signed, traceable.
|
||||
* **Air‑gap is a first‑class citizen.** Mirror bundles are not an afterthought; they’re how serious orgs actually run.
|
||||
|
||||
> Final reminder: **Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.**
|
||||
BIN
docs/implplan/EPIC_11.md
Normal file
BIN
docs/implplan/EPIC_11.md
Normal file
Binary file not shown.
BIN
docs/implplan/EPIC_12.md
Normal file
BIN
docs/implplan/EPIC_12.md
Normal file
Binary file not shown.
BIN
docs/implplan/EPIC_13.md
Normal file
BIN
docs/implplan/EPIC_13.md
Normal file
Binary file not shown.
BIN
docs/implplan/EPIC_14.md
Normal file
BIN
docs/implplan/EPIC_14.md
Normal file
Binary file not shown.
BIN
docs/implplan/EPIC_15.md
Normal file
BIN
docs/implplan/EPIC_15.md
Normal file
Binary file not shown.
1
docs/implplan/EPIC_16.md
Normal file
1
docs/implplan/EPIC_16.md
Normal file
@@ -0,0 +1 @@
|
||||
See `docs/airgap/EPIC_16_AIRGAP_MODE.md` for the full Epic 16 specification.
|
||||
1
docs/implplan/EPIC_17.md
Normal file
1
docs/implplan/EPIC_17.md
Normal file
@@ -0,0 +1 @@
|
||||
See `docs/api/EPIC_17_SDKS_OPENAPI.md` for the complete Epic 17 specification.
|
||||
1
docs/implplan/EPIC_18.md
Normal file
1
docs/implplan/EPIC_18.md
Normal file
@@ -0,0 +1 @@
|
||||
See `docs/risk/EPIC_18_RISK_PROFILES.md` for the complete Epic 18 specification.
|
||||
1
docs/implplan/EPIC_19.md
Normal file
1
docs/implplan/EPIC_19.md
Normal file
@@ -0,0 +1 @@
|
||||
See `docs/attestor/EPIC_19_ATTESTOR_CONSOLE.md` for the complete Epic 19 specification.
|
||||
567
docs/implplan/EPIC_2.md
Normal file
567
docs/implplan/EPIC_2.md
Normal file
@@ -0,0 +1,567 @@
|
||||
Fine. Here’s the next epic, written so you can paste it straight into the repo without having to babysit me. Same structure as before, maximum detail, zero hand‑waving.
|
||||
|
||||
---
|
||||
|
||||
# Epic 2: Policy Engine & Policy Editor (VEX + Advisory Application Rules)
|
||||
|
||||
> Short name: **Policy Engine v2**
|
||||
> Services touched: **Policy Engine, Web API, Console (Policy Editor), CLI, Conseiller, Excitator, SBOM Service, Authority, Workers/Scheduler**
|
||||
> Data stores: **MongoDB (policies, runs, effective findings), optional Redis/NATS for jobs**
|
||||
|
||||
---
|
||||
|
||||
## 1) What it is
|
||||
|
||||
This epic delivers the **organization‑specific decision layer** for Stella. Ingestion is now AOC‑compliant (Epic 1). That means advisories and VEX arrive as immutable raw facts. This epic builds the place where those facts become **effective findings** under policies you control.
|
||||
|
||||
Core deliverables:
|
||||
|
||||
* **Policy Engine**: deterministic evaluator that applies rule sets to inputs:
|
||||
|
||||
* Inputs: `advisory_raw`, `vex_raw`, SBOMs, optional telemetry hooks (reachability stubs), org metadata.
|
||||
* Outputs: `effective_finding_{policyId}` materializations, with full explanation traces.
|
||||
* **Policy Editor (Console + CLI)**: versioned policy authoring, simulation, review/approval workflow, and change diffs.
|
||||
* **Rules DSL v1**: safe, declarative language for VEX application, advisory normalization, and risk scoring. No arbitrary code execution, no network calls.
|
||||
* **Run Orchestrator**: incremental re‑evaluation when new raw facts or SBOM changes arrive; efficient partial updates.
|
||||
|
||||
The philosophy is boring on purpose: policy is a **pure function of inputs**. Same inputs and same policy yield the same outputs, every time, on every machine. If you want drama, watch reality TV, not your risk pipeline.
|
||||
|
||||
---
|
||||
|
||||
## 2) Why
|
||||
|
||||
* Vendors disagree, contexts differ, and your tolerance for risk is not universal.
|
||||
* VEX means nothing until you decide **how** to apply it to **your** assets.
|
||||
* Auditors care about the “why.” You’ll need consistent, replayable answers, with traces.
|
||||
* Security teams need **simulation** before rollouts, and **diffs** after.
|
||||
|
||||
---
|
||||
|
||||
## 3) How it should work (deep details)
|
||||
|
||||
### 3.1 Data model
|
||||
|
||||
#### 3.1.1 Policy documents (Mongo: `policies`)
|
||||
|
||||
```json
|
||||
{
|
||||
"_id": "policy:P-7:v3",
|
||||
"policy_id": "P-7",
|
||||
"version": 3,
|
||||
"name": "Default Org Policy",
|
||||
"status": "approved", // draft | submitted | approved | archived
|
||||
"owned_by": "team:sec-plat",
|
||||
"valid_from": "2025-01-15T00:00:00Z",
|
||||
"valid_to": null,
|
||||
"dsl": {
|
||||
"syntax": "stella-dsl@1",
|
||||
"source": "rule-set text or compiled IR ref"
|
||||
},
|
||||
"metadata": {
|
||||
"description": "Baseline scoring + VEX precedence",
|
||||
"tags": ["baseline","vex","cvss"]
|
||||
},
|
||||
"provenance": {
|
||||
"created_by": "user:ali",
|
||||
"created_at": "2025-01-15T08:00:00Z",
|
||||
"submitted_by": "user:kay",
|
||||
"approved_by": "user:root",
|
||||
"approval_at": "2025-01-16T10:00:00Z",
|
||||
"checksum": "sha256:..."
|
||||
},
|
||||
"tenant": "default"
|
||||
}
|
||||
```
|
||||
|
||||
Constraints:
|
||||
|
||||
* `status=approved` is required to run in production.
|
||||
* Version increments are append‑only. Old versions remain runnable for replay.
|
||||
|
||||
#### 3.1.2 Policy runs (Mongo: `policy_runs`)
|
||||
|
||||
```json
|
||||
{
|
||||
"_id": "run:P-7:2025-02-20T12:34:56Z:abcd",
|
||||
"policy_id": "P-7",
|
||||
"policy_version": 3,
|
||||
"inputs": {
|
||||
"sbom_set": ["sbom:S-42"],
|
||||
"advisory_cursor": "2025-02-20T00:00:00Z",
|
||||
"vex_cursor": "2025-02-20T00:00:00Z"
|
||||
},
|
||||
"mode": "incremental", // full | incremental | simulate
|
||||
"stats": {
|
||||
"components": 1742,
|
||||
"advisories_considered": 9210,
|
||||
"vex_considered": 1187,
|
||||
"rules_fired": 68023,
|
||||
"findings_out": 4321
|
||||
},
|
||||
"trace": {
|
||||
"location": "blob://traces/run-.../index.json",
|
||||
"sampling": "smart-10pct"
|
||||
},
|
||||
"status": "succeeded", // queued | running | failed | succeeded | canceled
|
||||
"started_at": "2025-02-20T12:34:56Z",
|
||||
"finished_at": "2025-02-20T12:35:41Z",
|
||||
"tenant": "default"
|
||||
}
|
||||
```
|
||||
|
||||
#### 3.1.3 Effective findings (Mongo: `effective_finding_P-7`)
|
||||
|
||||
```json
|
||||
{
|
||||
"_id": "P-7:S-42:pkg:npm/lodash@4.17.21:CVE-2021-23337",
|
||||
"policy_id": "P-7",
|
||||
"policy_version": 3,
|
||||
"sbom_id": "S-42",
|
||||
"component_purl": "pkg:npm/lodash@4.17.21",
|
||||
"advisory_ids": ["CVE-2021-23337", "GHSA-..."],
|
||||
"status": "affected", // affected | not_affected | fixed | under_investigation | suppressed
|
||||
"severity": {
|
||||
"normalized": "High",
|
||||
"score": 7.5,
|
||||
"vector": "CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:H/A:N",
|
||||
"rationale": "cvss_base(OSV) + vendor_weighting + env_modifiers"
|
||||
},
|
||||
"rationale": [
|
||||
{"rule":"vex.precedence","detail":"VendorX not_affected justified=component_not_present wins"},
|
||||
{"rule":"advisory.cvss.normalization","detail":"mapped GHSA severity to CVSS 3.1 = 7.5"}
|
||||
],
|
||||
"references": {
|
||||
"advisory_raw_ids": ["advisory_raw:osv:GHSA-...:v3"],
|
||||
"vex_raw_ids": ["vex_raw:VendorX:doc-123:v4"]
|
||||
},
|
||||
"run_id": "run:P-7:2025-02-20T12:34:56Z:abcd",
|
||||
"tenant": "default"
|
||||
}
|
||||
```
|
||||
|
||||
Write protection: only the **Policy Engine** service identity may write any `effective_finding_*` collection.
|
||||
|
||||
---
|
||||
|
||||
### 3.2 Rules DSL v1 (stella‑dsl@1)
|
||||
|
||||
**Design goals**
|
||||
|
||||
* Declarative, composable, deterministic.
|
||||
* No loops, no network IO, no non‑deterministic time.
|
||||
* Policy authors see readable text; the engine compiles to a safe IR.
|
||||
|
||||
**Concepts**
|
||||
|
||||
* **WHEN** condition matches a tuple `(sbom_component, advisory, optional vex_statements)`
|
||||
* **THEN** actions set `status`, compute `severity`, attach `rationale`, or `suppress` with reason.
|
||||
* **Profiles** for severity and scoring; **Maps** for vendor weighting; **Guards** for VEX justification.
|
||||
|
||||
**Mini‑grammar (subset)**
|
||||
|
||||
```
|
||||
policy "Default Org Policy" syntax "stella-dsl@1" {
|
||||
|
||||
profile severity {
|
||||
map vendor_weight {
|
||||
source "GHSA" => +0.5
|
||||
source "OSV" => +0.0
|
||||
source "VendorX" => -0.2
|
||||
}
|
||||
env base_cvss {
|
||||
if env.runtime == "serverless" then -0.5
|
||||
if env.exposure == "internal-only" then -1.0
|
||||
}
|
||||
}
|
||||
|
||||
rule vex_precedence {
|
||||
when vex.any(status in ["not_affected","fixed"])
|
||||
and vex.justification in ["component_not_present","vulnerable_code_not_present"]
|
||||
then status := vex.status
|
||||
because "VEX strong justification prevails";
|
||||
}
|
||||
|
||||
rule advisory_to_cvss {
|
||||
when advisory.source in ["GHSA","OSV"]
|
||||
then severity := normalize_cvss(advisory)
|
||||
because "Map vendor severity or CVSS vector";
|
||||
}
|
||||
|
||||
rule reachability_soft_suppress {
|
||||
when severity.normalized <= "Medium"
|
||||
and telemetry.reachability == "none"
|
||||
then status := "suppressed"
|
||||
because "not reachable and low severity";
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Built‑ins** (non‑exhaustive)
|
||||
|
||||
* `normalize_cvss(advisory)` maps GHSA/OSV/CSAF severity fields to CVSS v3.1 numbers when possible; otherwise vendor‑to‑numeric mapping table in policy.
|
||||
* `vex.any(...)` tests across matching VEX statements for the same `(component, advisory)`.
|
||||
* `telemetry.*` is an optional input namespace reserved for future reachability data; if absent, expressions evaluate to `unknown` (no effect).
|
||||
|
||||
**Determinism**
|
||||
|
||||
* Rules are evaluated in **stable order**: explicit `priority` attribute or lexical order.
|
||||
* **First‑match** semantics for conflicting status unless `combine` is used.
|
||||
* Severity computations are pure; numeric maps are part of policy document.
|
||||
|
||||
---
|
||||
|
||||
### 3.3 Evaluation model
|
||||
|
||||
1. **Selection**
|
||||
|
||||
* For each SBOM component PURL, find candidate advisories from `advisory_raw` via linkset PURLs or identifiers.
|
||||
* For each pair `(component, advisory)`, load all matching VEX facts from `vex_raw`.
|
||||
|
||||
2. **Context assembly**
|
||||
|
||||
* Build an evaluation context from:
|
||||
|
||||
* `sbom_component`: PURL, licenses, relationships.
|
||||
* `advisory`: source, identifiers, references, embedded vendor severity (kept in `content.raw`).
|
||||
* `vex`: list of statements with status and justification.
|
||||
* `env`: org‑specific env vars configured per policy run (e.g., exposure).
|
||||
* Optional `telemetry` if available.
|
||||
|
||||
3. **Rule execution**
|
||||
|
||||
* Compile DSL to IR once per policy version; cache.
|
||||
* Execute rules per tuple; record which rules fired and the order.
|
||||
* If no rule sets status, default is `affected`.
|
||||
* If no rule sets severity, default severity uses `normalize_cvss(advisory)` with vendor defaults.
|
||||
|
||||
4. **Materialization**
|
||||
|
||||
* Write to `effective_finding_{policyId}` with `rationale` chain and references to raw docs.
|
||||
* Emit per‑tuple trace events; sample and store full traces per run.
|
||||
|
||||
5. **Incremental updates**
|
||||
|
||||
* A watch job observes new `advisory_raw` and `vex_raw` inserts and SBOM deltas.
|
||||
* The orchestrator computes the affected tuples and re‑evaluates only those.
|
||||
|
||||
6. **Replay**
|
||||
|
||||
* Any `policy_run` is fully reproducible by `(policy_id, version, input set, cursors)`.
|
||||
|
||||
---
|
||||
|
||||
### 3.4 VEX application semantics
|
||||
|
||||
* **Precedence**: a `not_affected` with strong justification (`component_not_present`, `vulnerable_code_not_present`, `fix_not_required`) wins unless another rule explicitly overrides by environment context.
|
||||
* **Scoping**: VEX statements often specify product/component scope. Matching uses PURL equivalence and version ranges extracted during ingestion linkset generation.
|
||||
* **Conflicts**: If multiple VEX statements conflict, the default is **most‑specific scope wins** (component > product > vendor), then newest `document_version`. Policies can override with explicit rules.
|
||||
* **Explainability**: Every VEX‑driven decision records which statement IDs were considered and which one won.
|
||||
|
||||
---
|
||||
|
||||
### 3.5 Advisory normalization rules
|
||||
|
||||
* **Vendor severity mapping**: Map GHSA levels or CSAF product‑tree severities to CVSS‑like numeric bands via policy maps.
|
||||
* **CVSS vector use**: If a valid vector exists in `content.raw`, parse and compute; apply policy modifiers from `profile severity`.
|
||||
* **Temporal/environment modifiers**: Optional reductions for network exposure, isolation, or compensating controls, all encoded in policy.
|
||||
|
||||
---
|
||||
|
||||
### 3.6 Performance and scale
|
||||
|
||||
* Partition evaluation by SBOM ID and hash ranges of PURLs.
|
||||
* Pre‑index `advisory_raw.linkset.purls` and `vex_raw.linkset.purls` (already in Epic 1).
|
||||
* Use streaming iterators; avoid loading entire SBOM or advisory sets into memory.
|
||||
* Materialize only changed findings (diff‑aware writes).
|
||||
* Target: 100k components, 1M advisories considered, 5 minutes incremental SLA on commodity hardware.
|
||||
|
||||
---
|
||||
|
||||
### 3.7 Error codes
|
||||
|
||||
| Code | Meaning | HTTP |
|
||||
| ------------- | ----------------------------------------------------- | ---- |
|
||||
| `ERR_POL_001` | Policy syntax error | 400 |
|
||||
| `ERR_POL_002` | Policy not approved for run | 403 |
|
||||
| `ERR_POL_003` | Missing inputs (SBOM/advisory/vex fetch failed) | 424 |
|
||||
| `ERR_POL_004` | Determinism guard triggered (non‑pure function usage) | 500 |
|
||||
| `ERR_POL_005` | Write denied to effective findings (caller invalid) | 403 |
|
||||
| `ERR_POL_006` | Run canceled or timed out | 408 |
|
||||
|
||||
---
|
||||
|
||||
### 3.8 Observability
|
||||
|
||||
* Metrics:
|
||||
|
||||
* `policy_compile_seconds`, `policy_run_seconds{mode=...}`, `rules_fired_total`, `findings_written_total`, `vex_overrides_total`, `simulate_diff_total{delta=up|down|unchanged}`.
|
||||
* Tracing:
|
||||
|
||||
* Spans: `policy.compile`, `policy.select`, `policy.eval`, `policy.materialize`.
|
||||
* Logs:
|
||||
|
||||
* Include `policy_id`, `version`, `run_id`, `sbom_id`, `component_purl`, `advisory_id`, `vex_count`, `rule_hits`.
|
||||
|
||||
> **Imposed rule:** Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.
|
||||
|
||||
---
|
||||
|
||||
### 3.9 Security and tenancy
|
||||
|
||||
* Only users with `policy:write` can create/modify policies.
|
||||
* `policy:approve` is a separate privileged role.
|
||||
* Only Policy Engine service identity has `effective:write`.
|
||||
* Tenancy is explicit on all documents and queries.
|
||||
|
||||
> **Imposed rule:** Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.
|
||||
|
||||
---
|
||||
|
||||
## 4) API surface
|
||||
|
||||
### 4.1 Policy CRUD and lifecycle
|
||||
|
||||
* `POST /policies` create draft
|
||||
* `GET /policies?status=...` list
|
||||
* `GET /policies/{policyId}/versions/{v}` fetch
|
||||
* `POST /policies/{policyId}/submit` move draft to submitted
|
||||
* `POST /policies/{policyId}/approve` approve version
|
||||
* `POST /policies/{policyId}/archive` archive version
|
||||
|
||||
### 4.2 Compilation and validation
|
||||
|
||||
* `POST /policies/{policyId}/versions/{v}/compile`
|
||||
|
||||
* Returns IR checksum, syntax diagnostics, rule stats.
|
||||
|
||||
### 4.3 Runs
|
||||
|
||||
* `POST /policies/{policyId}/runs` body: `{mode, sbom_set, advisory_cursor?, vex_cursor?, env?}`
|
||||
* `GET /policies/{policyId}/runs/{runId}` status + stats
|
||||
* `POST /policies/{policyId}/simulate` returns **diff** vs current approved version on a sample SBOM set.
|
||||
|
||||
### 4.4 Findings and explanations
|
||||
|
||||
* `GET /findings/{policyId}?sbom_id=S-42&status=affected&severity=High+Critical`
|
||||
* `GET /findings/{policyId}/{findingId}/explain` returns ordered rule hits and linked raw IDs.
|
||||
|
||||
All endpoints require tenant scoping and appropriate `policy:*` or `findings:*` roles.
|
||||
|
||||
---
|
||||
|
||||
## 5) Console (Policy Editor) and CLI behavior
|
||||
|
||||
**Console**
|
||||
|
||||
* Monaco‑style editor with DSL syntax highlighting, lint, quick docs.
|
||||
* Side‑by‑side **Simulation** panel: show count of affected findings before/after.
|
||||
* Approval workflow: submit, review comments, approve with rationale.
|
||||
* Diffs: show rule‑wise changes and estimated impact.
|
||||
* Read‑only run viewer: heatmap of rules fired, top suppressions, VEX wins.
|
||||
|
||||
**CLI**
|
||||
|
||||
* `stella policy new --name "Default Org Policy"`
|
||||
* `stella policy edit P-7` opens local editor -> `submit`
|
||||
* `stella policy approve P-7 --version 3`
|
||||
* `stella policy simulate P-7 --sbom S-42 --env exposure=internal-only`
|
||||
* `stella findings ls --policy P-7 --sbom S-42 --status affected`
|
||||
|
||||
Exit codes map to `ERR_POL_*`.
|
||||
|
||||
---
|
||||
|
||||
## 6) Implementation tasks
|
||||
|
||||
### 6.1 Policy Engine service
|
||||
|
||||
* [ ] Implement DSL parser and IR compiler (`stella-dsl@1`).
|
||||
* [ ] Build evaluator with stable ordering and first‑match semantics.
|
||||
* [ ] Implement selection joiners for SBOM↔advisory↔vex using linksets.
|
||||
* [ ] Materialization writer with upsert‑only semantics to `effective_finding_{policyId}`.
|
||||
* [ ] Determinism guard (ban wall‑clock, network, and RNG during eval).
|
||||
* [ ] Incremental orchestrator listening to advisory/vex/SBOM change streams.
|
||||
* [ ] Trace emitter with rule‑hit sampling.
|
||||
* [ ] Unit tests, property tests, golden fixtures; perf tests to target SLA.
|
||||
|
||||
**Imposed rule:** Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.
|
||||
|
||||
### 6.2 Web API
|
||||
|
||||
* [ ] Policy CRUD, compile, run, simulate, findings, explain endpoints.
|
||||
* [ ] Pagination, filters, and tenant enforcement on all list endpoints.
|
||||
* [ ] Error mapping to `ERR_POL_*`.
|
||||
* [ ] Rate limits on simulate endpoints.
|
||||
|
||||
**Imposed rule:** Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.
|
||||
|
||||
### 6.3 Console (Policy Editor)
|
||||
|
||||
* [ ] Editor with DSL syntax highlighting and inline diagnostics.
|
||||
* [ ] Simulation UI with pre/post counts and top deltas.
|
||||
* [ ] Approval workflow UI with audit trail.
|
||||
* [ ] Run viewer dashboards (rule heatmap, VEX wins, suppressions).
|
||||
|
||||
**Imposed rule:** Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.
|
||||
|
||||
### 6.4 CLI
|
||||
|
||||
* [ ] New commands: `policy new|edit|submit|approve|simulate`, `findings ls|get`.
|
||||
* [ ] Json/YAML output formats for CI consumption.
|
||||
* [ ] Non‑zero exits on syntax errors or simulation failures; map to `ERR_POL_*`.
|
||||
|
||||
**Imposed rule:** Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.
|
||||
|
||||
### 6.5 Conseiller & Excitator integration
|
||||
|
||||
* [ ] Provide search endpoints optimized for policy selection (batch by PURLs and IDs).
|
||||
* [ ] Harden linkset extraction to maximize join recall.
|
||||
* [ ] Add cursors for incremental selection windows per run.
|
||||
|
||||
**Imposed rule:** Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.
|
||||
|
||||
### 6.6 SBOM Service
|
||||
|
||||
* [ ] Ensure fast PURL index and component metadata projection for policy queries.
|
||||
* [ ] Provide relationship graph API for future transitive logic.
|
||||
* [ ] Emit change events on SBOM updates.
|
||||
|
||||
**Imposed rule:** Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.
|
||||
|
||||
### 6.7 Authority
|
||||
|
||||
* [ ] Define scopes: `policy:write`, `policy:approve`, `policy:run`, `findings:read`, `effective:write`.
|
||||
* [ ] Issue service identity for Policy Engine with `effective:write` only.
|
||||
* [ ] Enforce tenant claims at gateway.
|
||||
|
||||
**Imposed rule:** Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.
|
||||
|
||||
### 6.8 CI/CD
|
||||
|
||||
* [ ] Lint policy DSL in PRs; block invalid syntax.
|
||||
* [ ] Run `simulate` against golden SBOMs to detect explosive deltas.
|
||||
* [ ] Determinism CI: two runs with identical seeds produce identical outputs.
|
||||
|
||||
**Imposed rule:** Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.
|
||||
|
||||
---
|
||||
|
||||
## 7) Documentation changes (create/update these files)
|
||||
|
||||
1. **`/docs/policy/overview.md`**
|
||||
|
||||
* What the Policy Engine is, high‑level concepts, inputs, outputs, determinism.
|
||||
2. **`/docs/policy/dsl.md`**
|
||||
|
||||
* Full grammar, built‑ins, examples, best practices, anti‑patterns.
|
||||
3. **`/docs/policy/lifecycle.md`**
|
||||
|
||||
* Draft → submitted → approved → archived, roles, and audit trail.
|
||||
4. **`/docs/policy/runs.md`**
|
||||
|
||||
* Run modes, incremental mechanics, cursors, replay.
|
||||
5. **`/docs/api/policy.md`**
|
||||
|
||||
* Endpoints, request/response schemas, error codes.
|
||||
6. **`/docs/cli/policy.md`**
|
||||
|
||||
* Command usage, examples, exit codes, JSON output contracts.
|
||||
7. **`/docs/ui/policy-editor.md`**
|
||||
|
||||
* Screens, workflows, simulation, diffs, approvals.
|
||||
8. **`/docs/architecture/policy-engine.md`**
|
||||
|
||||
* Detailed sequence diagrams, selection/join strategy, materialization schema.
|
||||
9. **`/docs/observability/policy.md`**
|
||||
|
||||
* Metrics, tracing, logs, sample dashboards.
|
||||
10. **`/docs/security/policy-governance.md`**
|
||||
|
||||
* Scopes, approvals, tenancy, least privilege.
|
||||
11. **`/docs/examples/policies/`**
|
||||
|
||||
* `baseline.pol`, `serverless.pol`, `internal-only.pol`, each with commentary.
|
||||
12. **`/docs/faq/policy-faq.md`**
|
||||
|
||||
* Common pitfalls, VEX conflict handling, determinism gotchas.
|
||||
|
||||
Each file includes a **Compliance checklist** for authors and reviewers.
|
||||
|
||||
> **Imposed rule:** Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.
|
||||
|
||||
---
|
||||
|
||||
## 8) Acceptance criteria
|
||||
|
||||
* Policies are versioned, approvable, and compilable; invalid DSL blocks merges.
|
||||
* Engine produces deterministic outputs with full rationale chains.
|
||||
* VEX precedence rules work per spec and are overridable by policy.
|
||||
* Simulation yields accurate pre/post deltas and diffs.
|
||||
* Only Policy Engine can write to `effective_finding_*`.
|
||||
* Incremental runs pick up new advisories/VEX/SBOM changes without full re‑runs.
|
||||
* Console and CLI cover authoring, simulation, approval, and retrieval.
|
||||
* Observability dashboards show rule hits, VEX wins, and run timings.
|
||||
|
||||
---
|
||||
|
||||
## 9) Risks and mitigations
|
||||
|
||||
* **Policy sprawl**: too many similar policies.
|
||||
|
||||
* Mitigation: templates, policy inheritance in v1.1, tagging, ownership metadata.
|
||||
* **Non‑determinism creep**: someone sneaks wall‑clock or network into evaluation.
|
||||
|
||||
* Mitigation: determinism guard, static analyzer, and CI replay check.
|
||||
* **Join miss‑rate**: weak linksets cause under‑matching.
|
||||
|
||||
* Mitigation: linkset strengthening in ingestion, PURL equivalence tables, monitoring for “zero‑hit” rates.
|
||||
* **Approval bottlenecks**: blocked rollouts.
|
||||
|
||||
* Mitigation: RBAC with delegated approvers and time‑boxed SLAs.
|
||||
|
||||
---
|
||||
|
||||
## 10) Test plan
|
||||
|
||||
* **Unit**: parser, compiler, evaluator; conflict resolution; precedence.
|
||||
* **Property**: random policies over synthetic inputs; ensure no panics and stable outputs.
|
||||
* **Golden**: fixed SBOM + curated advisories/VEX → expected findings; compare every run.
|
||||
* **Performance**: large SBOMs with heavy rule sets; assert run times and memory ceilings.
|
||||
* **Integration**: end‑to‑end simulate → approve → run → diff; verify write protections.
|
||||
* **Chaos**: inject malformed VEX, missing advisories; ensure graceful degradation and clear errors.
|
||||
|
||||
---
|
||||
|
||||
## 11) Developer checklists
|
||||
|
||||
**Definition of Ready**
|
||||
|
||||
* Policy grammar finalized; examples prepared.
|
||||
* Linkset join queries benchmarked.
|
||||
* Owner and approvers assigned.
|
||||
|
||||
**Definition of Done**
|
||||
|
||||
* All APIs live with RBAC.
|
||||
* CLI and Console features shipped.
|
||||
* Determinism and golden tests green.
|
||||
* Observability dashboards deployed.
|
||||
* Docs in section 7 merged.
|
||||
* Two real org policies migrated and in production.
|
||||
|
||||
---
|
||||
|
||||
## 12) Glossary
|
||||
|
||||
* **Policy**: versioned rule set controlling status and severity.
|
||||
* **DSL**: domain‑specific language used to express rules.
|
||||
* **Run**: a single evaluation execution with defined inputs and outputs.
|
||||
* **Simulation**: a run that doesn’t write findings; returns diffs.
|
||||
* **Materialization**: persisted effective findings for fast queries.
|
||||
* **Determinism**: same inputs + same policy = same outputs. Always.
|
||||
|
||||
---
|
||||
|
||||
### Final imposed reminder
|
||||
|
||||
**Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.**
|
||||
409
docs/implplan/EPIC_4.md
Normal file
409
docs/implplan/EPIC_4.md
Normal file
@@ -0,0 +1,409 @@
|
||||
Here’s Epic 4 in the same paste‑into‑repo, implementation‑ready style as the prior epics. It’s exhaustive, formal, and slots directly into the existing AOC model, Policy Engine, and Console.
|
||||
|
||||
---
|
||||
|
||||
# Epic 4: Policy Studio (author, version, simulate)
|
||||
|
||||
> Short name: **Policy Studio**
|
||||
> Services touched: **Policy Engine**, **Policy Registry** (new), **Web API Gateway**, **Authority** (authN/Z), **Scheduler/Workers**, **SBOM Service**, **Conseiller (Feedser)**, **Excitator (Vexer)**, **Telemetry**
|
||||
> Surfaces: **Console (Web UI)** feature module, **CLI**, **CI hooks**
|
||||
> Deliverables: Authoring workspace, policy versioning, static checks, simulation at scale, reviews/approvals, signing/publishing, promotion
|
||||
|
||||
---
|
||||
|
||||
## 1) What it is
|
||||
|
||||
**Policy Studio** is the end‑to‑end system for creating, evolving, and safely rolling out the rules that turn AOC facts (SBOM, advisories, VEX) into **effective findings**. It provides:
|
||||
|
||||
* A **workspace** where authors write policies in the DSL (Epic 2), with linting, autocompletion, snippets, and templates.
|
||||
* A **Policy Registry** that stores immutable versions, compiled artifacts, metadata, provenance, and signatures.
|
||||
* **Simulation** at two levels: quick local samples and large batch simulations across real SBOM inventories with full deltas.
|
||||
* A **review/approval** workflow with comments, diffs, required approvers, and promotion to environments (dev/test/prod).
|
||||
* **Publishing** semantics: signed, immutable versions bound to tenants; rollback and deprecation.
|
||||
* Tight integration with **Explain** traces so any change can show exactly which rules fired and why outcomes shifted.
|
||||
|
||||
The Studio respects **AOC enforcement**: policies never edit or merge facts. They only interpret facts and produce determinations consistent with precedence rules defined in the DSL.
|
||||
|
||||
---
|
||||
|
||||
## 2) Why
|
||||
|
||||
* Policy errors are expensive. Authors need safe sandboxes, deterministic builds, and evidence before rollout.
|
||||
* Auditors require immutability, provenance, and reproducibility from “source policy” to “effective finding.”
|
||||
* Teams want gradual rollout: simulate, canary, promote, observe, rollback.
|
||||
* Policy knowledge should be modular, reusable, and testable, not tribal.
|
||||
|
||||
---
|
||||
|
||||
## 3) How it should work (maximum detail)
|
||||
|
||||
### 3.1 Domain model
|
||||
|
||||
* **PolicyPackage**: `{name, tenant, description, owners[], tags[], created_at}`
|
||||
* **PolicyVersion** (immutable): `{package, semver, source_sha, compiled_sha, status: draft|review|approved|published|deprecated|archived, created_by, created_at, signatures[], changelog, metadata{}}`
|
||||
* **Workspace**: mutable working area for authors; holds unversioned edits until compiled.
|
||||
* **CompilationArtifact**: `{policy_version, compiler_version, diagnostics[], rule_index[], symbol_table}`
|
||||
* **SimulationSpec**: `{policy_version|workspace, sbom_selector, time_window?, environment?, sample_size?, severity_floor?, includes{advisories?, vex?}}`
|
||||
* **SimulationRun**: `{run_id, spec, started_at, finished_at, result{counts_before, counts_after, top_deltas[], by_rule_hit[], sample_explains[]}}`
|
||||
* **Review**: `{policy_version, required_approvers[], votes[], comments[], files_changed[], diffs[]}`
|
||||
* **Promotion**: `{policy_version, environment: dev|test|prod, promoted_by, promoted_at, rollout_strategy: All|Percent|TenantSubset}`
|
||||
* **Attestation**: OIDC‑backed signature metadata binding `source_sha` and `compiled_sha` to an actor and time.
|
||||
|
||||
> **Imposed rule:** Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.
|
||||
|
||||
### 3.2 Authoring workflow
|
||||
|
||||
1. **Create** a workspace from a template (e.g., “Default Risk Model,” “License Tilted,” “Cloud‑Native SBOM”).
|
||||
2. **Edit** in the Studio: Monaco editor with DSL grammar, intelligent completion for predicates, policies, attributes.
|
||||
3. **Lint & compile** locally: semantic checks, forbidden rules detection, policy size limits, constant‑folding.
|
||||
4. **Unit tests**: run policy test cases on bundled fixtures and golden expectations.
|
||||
5. **Quick simulate** on selected SBOMs (10–50 items) to preview counts, examples, and rule heatmap.
|
||||
6. **Propose version**: bump semver, enter changelog; create a **PolicyVersion** in `review` with compiled artifacts.
|
||||
7. **Review & approval**: side‑by‑side diff, comments, required approvers enforced by RBAC.
|
||||
8. **Batch simulation**: run at scale across tenant inventory; produce deltas, sample explainer evidence.
|
||||
9. **Publish**: sign and move to `published`; optional **Promotion** to target environment(s).
|
||||
10. **Run** evaluation with the selected policy version; verify outcomes; optionally promote to default.
|
||||
11. **Rollback**: select an older version; promotion updates references without mutating older versions.
|
||||
|
||||
### 3.3 Editing experience (Console)
|
||||
|
||||
* **Three‑pane layout**: file tree, editor, diagnostics/simulation.
|
||||
* **Features**: autocomplete from symbol table, in‑editor docs on hover, go‑to definition, rule references, rename symbols across files, snippet library, policy templates.
|
||||
* **Validations**:
|
||||
|
||||
* AOC guardrails: no edit/merge actions on source facts, only interpretation.
|
||||
* Precedence correctness: if rules conflict, studio shows explicit order and effective winner.
|
||||
* Severity floor and normalization mapping validated against registry configuration.
|
||||
* **Diagnostics panel**: errors, warnings, performance hints (e.g., “predicate X loads N advisories per component; consider indexing”).
|
||||
* **Rule heatmap**: during simulation, bar chart of rule firings and the objects they impact.
|
||||
* **Explain sampler**: click any delta bucket to open a sampled finding with full trace.
|
||||
|
||||
### 3.4 Simulation
|
||||
|
||||
* **Quick Sim**: synchronous; runs in browser‑orchestrated job against API, constrained by `sample_size`.
|
||||
* **Batch Sim**: asynchronous run in workers:
|
||||
|
||||
* Input selection: all SBOMs, labels, artifact regex, last N ingests, or a curated set.
|
||||
* Outputs: counts by severity before/after, by status, top deltas by component and advisory, rule heatmap, top K affected artifacts.
|
||||
* Evidence: NDJSON of sampled findings with traces; CSV summary; signed result manifest.
|
||||
* Guardrails: cannot publish if batch sim drift > configurable threshold without an override justification.
|
||||
|
||||
### 3.5 Versioning & promotion
|
||||
|
||||
* Semver enforced: `major` implies compatibility break (e.g., precedence changes), `minor` adds rules, `patch` fixes.
|
||||
* **Immutable**: after `published`, the version cannot change; deprecate instead.
|
||||
* **Environment bindings**: dev/test/prod mapping per tenant; default policy per environment.
|
||||
* **Canary**: promote to a subset of tenants or artifacts; the Runs page displays A/B comparisons.
|
||||
|
||||
### 3.6 Review & approval
|
||||
|
||||
* Require N approvers by role; self‑approval optionally prohibited.
|
||||
* Line and file comments; overall decision with justification.
|
||||
* Review snapshot captures: diffs, diagnostics, simulation summary.
|
||||
* Webhooks to notify external systems of review events.
|
||||
|
||||
### 3.7 RBAC (Authority)
|
||||
|
||||
Roles per tenant:
|
||||
|
||||
* **Policy Author**: create/edit workspace, quick sim, propose versions.
|
||||
* **Policy Reviewer**: comment, request changes, approve/reject.
|
||||
* **Policy Approver**: final approve, publish.
|
||||
* **Policy Operator**: promote, rollback, schedule runs.
|
||||
* **Read‑only Auditor**: view everything, download evidence.
|
||||
|
||||
All actions server‑checked; UI only hides affordances.
|
||||
|
||||
### 3.8 CLI + CI integration
|
||||
|
||||
CLI verbs (examples):
|
||||
|
||||
```
|
||||
stella policy init --template default
|
||||
stella policy lint
|
||||
stella policy compile
|
||||
stella policy test --golden ./tests
|
||||
stella policy simulate --sboms label:prod --sample 1000
|
||||
stella policy version bump --level minor --changelog "Normalize GHSA CVSS"
|
||||
stella policy submit --reviewers alice@example.com,bob@example.com
|
||||
stella policy approve --version 1.3.0
|
||||
stella policy publish --version 1.3.0 --sign
|
||||
stella policy promote --version 1.3.0 --env test --percent 20
|
||||
stella policy rollback --env prod --to 1.2.1
|
||||
```
|
||||
|
||||
CI usage:
|
||||
|
||||
* Lint, compile, and run unit tests on PRs that modify `/policies/**`.
|
||||
* Optionally trigger **Batch Sim** against a staging inventory and post a Markdown report to the PR.
|
||||
* Block merge if diagnostics include errors or drift exceeds thresholds.
|
||||
|
||||
> **Imposed rule:** Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.
|
||||
|
||||
### 3.9 APIs (representative)
|
||||
|
||||
* `POST /policies/workspaces` create from template
|
||||
* `PUT /policies/workspaces/{id}/files` edit source files
|
||||
* `POST /policies/workspaces/{id}/compile` get diagnostics + compiled artifact
|
||||
* `POST /policies/workspaces/{id}/simulate` quick sim
|
||||
* `POST /policies/versions` create version from workspace with semver + changelog
|
||||
* `GET /policies/versions/{id}` fetch version + diagnostics + sim summary
|
||||
* `POST /policies/versions/{id}/reviews` open review
|
||||
* `POST /policies/versions/{id}/approve` record approval
|
||||
* `POST /policies/versions/{id}/publish` sign + publish
|
||||
* `POST /policies/versions/{id}/promote` bind to env/canary
|
||||
* `POST /policies/versions/{id}/simulate-batch` start batch sim (async)
|
||||
* `GET /policies/simulations/{run_id}` get sim results and artifacts
|
||||
* `GET /policies/registry` list packages/versions, status and bindings
|
||||
|
||||
All calls require tenant scoping and RBAC.
|
||||
|
||||
### 3.10 Storage & data
|
||||
|
||||
* **Policy Registry DB** (MongoDB): packages, versions, workspaces, metadata.
|
||||
* **Object storage**: source bundles, compiled artifacts, simulation result bundles, evidence.
|
||||
* **Indexing**: compound indexes by `{tenant, package}`, `{tenant, status}`, `{tenant, environment}`.
|
||||
* **Retention**: configurable retention for workspaces and simulation artifacts; versions never deleted, only archived.
|
||||
|
||||
### 3.11 Evidence & provenance
|
||||
|
||||
* Every published version has:
|
||||
|
||||
* `source_sha` (content digest of the policy source bundle)
|
||||
* `compiled_sha` (digest of compiled artifact)
|
||||
* Attestation: signed envelope binding digests to an identity, time, and tenant.
|
||||
* Links to the exact compiler version, inputs, and environment.
|
||||
|
||||
### 3.12 Observability
|
||||
|
||||
* Metrics: compile time, diagnostics rate, simulation queue depth, delta magnitude distribution, approval latencies.
|
||||
* Logs: structured events for lifecycle transitions.
|
||||
* Traces: long simulations emit span per shard.
|
||||
|
||||
### 3.13 Performance & scale
|
||||
|
||||
* Compilation should complete under 3 seconds for typical policies; warn at 10s.
|
||||
* Batch sim uses workers with partitioning by SBOM id; results reduced by the API.
|
||||
* Memory guardrails on rule execution; deny policies that exceed configured complexity limits.
|
||||
|
||||
### 3.14 Security
|
||||
|
||||
* OIDC‑backed signing and attestation.
|
||||
* Policy sources are scanned on upload for secrets; blocked if found.
|
||||
* Strict CSP in Studio pages; tokens stored in memory, not localStorage.
|
||||
* Tenant isolation in buckets and DB collections.
|
||||
|
||||
> **Imposed rule:** Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.
|
||||
|
||||
---
|
||||
|
||||
## 4) Implementation plan
|
||||
|
||||
### 4.1 Services
|
||||
|
||||
* **Policy Registry (new microservice)**
|
||||
|
||||
* REST API and background workers for batch simulation orchestration.
|
||||
* Stores workspaces, versions, metadata, bindings, reviews.
|
||||
* Generates signed attestations at publish time.
|
||||
* Coordinates with **Policy Engine** for compile/simulate invocations.
|
||||
|
||||
* **Policy Engine (existing)**
|
||||
|
||||
* Expose compile and simulate endpoints with deterministic outputs.
|
||||
* Provide rule coverage, symbol table, and explain traces for samples.
|
||||
|
||||
* **Web API Gateway**
|
||||
|
||||
* Routes requests; injects tenant context; enforces RBAC.
|
||||
|
||||
### 4.2 Console (Web UI) feature module
|
||||
|
||||
* `packages/features/policies` (shared with Epic 3):
|
||||
|
||||
* **Studio** routes: `/policies/studio`, `/policies/:id/versions/:v/edit`, `/simulate`, `/review`.
|
||||
* Monaco editor wrapper for DSL with hover docs, autocomplete.
|
||||
* Diff viewer, diagnostics, heatmap, explain sampler, review UI.
|
||||
|
||||
### 4.3 CLI
|
||||
|
||||
* New commands under `stella policy *`; typed client generated from OpenAPI.
|
||||
* Outputs machine‑readable JSON and pretty tables.
|
||||
|
||||
### 4.4 Workers
|
||||
|
||||
* **Simulation workers**: pull shards of SBOMs, run policy, emit partials, reduce into result bundle.
|
||||
* **Notification worker**: sends webhooks on review, approval, publish, promote.
|
||||
|
||||
---
|
||||
|
||||
## 5) Documentation changes (create/update)
|
||||
|
||||
1. **`/docs/policy/studio-overview.md`**
|
||||
|
||||
* Concepts, roles, lifecycle, glossary.
|
||||
2. **`/docs/policy/authoring.md`**
|
||||
|
||||
* Workspace, templates, snippets, lint rules, best practices.
|
||||
3. **`/docs/policy/versioning-and-publishing.md`**
|
||||
|
||||
* Semver, immutability, deprecation, rollback, attestations.
|
||||
4. **`/docs/policy/simulation.md`**
|
||||
|
||||
* Quick vs batch sim, selection strategies, thresholds, evidence artifacts.
|
||||
5. **`/docs/policy/review-and-approval.md`**
|
||||
|
||||
* Required approvers, comments, webhooks, audit trail.
|
||||
6. **`/docs/policy/promotion.md`**
|
||||
|
||||
* Environments, canary, default policy binding, rollback.
|
||||
7. **`/docs/policy/cli.md`**
|
||||
|
||||
* Command reference with examples and JSON outputs.
|
||||
8. **`/docs/policy/api.md`**
|
||||
|
||||
* REST endpoints, request/response schemas, error codes.
|
||||
9. **`/docs/security/policy-attestations.md`**
|
||||
|
||||
* Signatures, digests, verifier steps.
|
||||
10. **`/docs/architecture/policy-registry.md`**
|
||||
|
||||
* Service design, schemas, queues, failure modes.
|
||||
11. **`/docs/observability/policy-telemetry.md`**
|
||||
|
||||
* Metrics, logs, tracing, dashboards.
|
||||
12. **`/docs/runbooks/policy-incident.md`**
|
||||
|
||||
* Rolling back a bad policy, freezing publishes, forensic steps.
|
||||
13. **`/docs/examples/policy-templates.md`**
|
||||
|
||||
* Ready‑made templates and snippet catalog.
|
||||
14. **`/docs/aoc/aoc-guardrails.md`**
|
||||
|
||||
* How Studio enforces AOC in authoring and review.
|
||||
|
||||
Each doc ends with a “Compliance checklist.”
|
||||
**Imposed rule:** Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.
|
||||
|
||||
---
|
||||
|
||||
## 6) Tasks
|
||||
|
||||
### 6.1 Backend: Policy Registry
|
||||
|
||||
* [ ] Define OpenAPI spec for Registry (workspaces, versions, reviews, sim).
|
||||
* [ ] Implement workspace storage and file CRUD.
|
||||
* [ ] Integrate with Policy Engine compile endpoint; return diagnostics, symbol table.
|
||||
* [ ] Implement quick simulation with request limits.
|
||||
* [ ] Implement batch simulation orchestration: enqueue shards, collect results, reduce deltas, store artifacts.
|
||||
* [ ] Implement review model: comments, required approvers, decisions.
|
||||
* [ ] Implement publish: sign, persist attestation, set status=published.
|
||||
* [ ] Implement promotion bindings per tenant/environment; canary subsets.
|
||||
* [ ] RBAC checks for all endpoints.
|
||||
* [ ] Unit/integration tests; load tests for batch sim.
|
||||
|
||||
### 6.2 Policy Engine enhancements
|
||||
|
||||
* [ ] Return rule coverage and firing counts with compile/simulate.
|
||||
* [ ] Return symbol table and inline docs for editor autocomplete.
|
||||
* [ ] Expose deterministic Explain traces for sampled findings.
|
||||
* [ ] Enforce complexity/time limits and report breaches.
|
||||
|
||||
### 6.3 Console (Web UI)
|
||||
|
||||
* [ ] Build Studio editor wrapper with Monaco + DSL language server hooks.
|
||||
* [ ] Implement file tree, snippets, templates, hotkeys, search/replace.
|
||||
* [ ] Diagnostics panel with jump‑to‑line, quick fixes.
|
||||
* [ ] Simulation panel: quick sim UI, charts, heatmap, sample explains.
|
||||
* [ ] Review UI: diff, comments, approvals, status badges.
|
||||
* [ ] Publish & Promote flows with confirmation and post‑actions.
|
||||
* [ ] Batch sim results pages with export buttons.
|
||||
* [ ] Accessibility audits and keyboard‑only authoring flow.
|
||||
|
||||
### 6.4 CLI
|
||||
|
||||
* [ ] Implement commands listed in 3.8 with rich help and examples.
|
||||
* [ ] Add `--json` flag for machine consumption; emit stable schemas.
|
||||
* [ ] Exit codes aligned with CI usage (lint errors → non‑zero).
|
||||
|
||||
### 6.5 CI/CD & Security
|
||||
|
||||
* [ ] Add CI job that runs `stella policy lint/compile/test` on PRs.
|
||||
* [ ] Optional job that triggers batch sim against staging inventory; post summary to PR.
|
||||
* [ ] Policy source secret scanning; block on findings.
|
||||
* [ ] Signing keys configuration; verify pipeline for attestation on publish.
|
||||
|
||||
### 6.6 Docs
|
||||
|
||||
* [ ] Write all docs in section 5 with screenshots and CLI transcripts.
|
||||
* [ ] Add cookbook examples and templates in `/docs/examples/policy-templates.md`.
|
||||
* [ ] Wire contextual Help links from Studio to relevant docs.
|
||||
|
||||
> **Imposed rule:** Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.
|
||||
|
||||
---
|
||||
|
||||
## 7) Acceptance criteria
|
||||
|
||||
* Authors can create, edit, lint, compile policies with inline diagnostics and autocomplete.
|
||||
* Quick simulation produces counts, rule heatmap, and sample explains within UI.
|
||||
* Batch simulation scales across large SBOM sets, producing deltas and downloadable evidence.
|
||||
* Review requires configured approvers; comments and diffs are preserved.
|
||||
* Publish generates immutable, signed versions with attestations.
|
||||
* Promotion binds versions to environments and supports canary and rollback.
|
||||
* CLI supports full lifecycle and is usable in CI.
|
||||
* All actions are tenant‑scoped, RBAC‑enforced, and logged.
|
||||
* AOC guardrails prevent any mutation of raw facts.
|
||||
* Documentation shipped and linked contextually from the Studio.
|
||||
|
||||
---
|
||||
|
||||
## 8) Risks & mitigations
|
||||
|
||||
* **Policy complexity causes timeouts** → compile‑time complexity scoring, execution limits, early diagnostics.
|
||||
* **Simulation cost at scale** → sharding and streaming reducers; sampling; configurable quotas.
|
||||
* **RBAC misconfiguration** → server‑enforced checks, defense‑in‑depth tests, deny‑by‑default.
|
||||
* **Attestation key management** → OIDC‑backed signatures; auditable verifier tool; time‑boxed credentials.
|
||||
* **Editor usability** → language server with accurate completions; docs on hover; snippet library.
|
||||
|
||||
---
|
||||
|
||||
## 9) Test plan
|
||||
|
||||
* **Unit**: compiler adapters, registry models, reviewers workflow, CLI options.
|
||||
* **Integration**: compile→simulate→publish→promote on seeded data.
|
||||
* **E2E**: Playwright flows for author→review→batch sim→publish→promote→rollback.
|
||||
* **Performance**: load test batch simulation with 100k components spread across SBOMs.
|
||||
* **Security**: RBAC matrix tests; secret scanning; signing and verification.
|
||||
* **Determinism**: same inputs produce identical `compiled_sha` and simulation summaries.
|
||||
|
||||
---
|
||||
|
||||
## 10) Feature flags
|
||||
|
||||
* `policy.studio` (enables editor and quick sim)
|
||||
* `policy.batch-sim`
|
||||
* `policy.canary-promotion`
|
||||
* `policy.signature-required` (enforce signing on publish)
|
||||
|
||||
Flags documented in `/docs/observability/policy-telemetry.md`.
|
||||
|
||||
---
|
||||
|
||||
## 11) Non‑goals (this epic)
|
||||
|
||||
* Building a general IDE for arbitrary languages; the editor is purpose‑built for the DSL.
|
||||
* Auto‑generated policies from AI without human approval.
|
||||
* Cross‑tenant policies; all policies are tenant‑scoped.
|
||||
|
||||
---
|
||||
|
||||
## 12) Philosophy
|
||||
|
||||
* **Safety first**: it’s cheaper to prevent a bad policy than to fix its fallout.
|
||||
* **Determinism**: same inputs, same outputs, verifiably.
|
||||
* **Immutability**: versions and evidence are forever; we deprecate, not mutate.
|
||||
* **Transparency**: every change is explainable with traces and proofs.
|
||||
* **Reusability**: templates, snippets, and tests turn policy from art into engineering.
|
||||
|
||||
> Final reminder: **Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.**
|
||||
431
docs/implplan/EPIC_5.md
Normal file
431
docs/implplan/EPIC_5.md
Normal file
@@ -0,0 +1,431 @@
|
||||
Here’s Epic 5 in the same paste‑into‑repo, implementation‑ready format as the prior epics. It’s exhaustive, formal, and designed to slot into AOC, Policy Engine, Conseiller/Excitator, and the Console.
|
||||
|
||||
---
|
||||
|
||||
# Epic 5: SBOM Graph Explorer
|
||||
|
||||
> Short name: **Graph Explorer**
|
||||
> Services touched: **SBOM Service**, **Graph Indexer** (new), **Graph API** (new), **Policy Engine**, **Conseiller (Feedser)**, **Excitator (Vexer)**, **Web API Gateway**, **Authority** (authN/Z), **Workers/Scheduler**, **Telemetry**
|
||||
> Surfaces: **Console (Web UI)** graph module, **CLI**, **Exports**
|
||||
> Deliverables: Interactive graph UI with semantic zoom, saved queries, policy/VEX/advisory overlays, diff views, impact analysis, exports
|
||||
|
||||
---
|
||||
|
||||
## 1) What it is
|
||||
|
||||
**SBOM Graph Explorer** is the interactive, tenant‑scoped view of all supply‑chain relationships the platform knows about, rendered as a navigable graph. It connects:
|
||||
|
||||
* **Artifacts** (applications, images, libs), **Packages/Versions**, **Files/Paths**, **Licenses**, **Advisories** (from Conseiller), **VEX statements** (from Excitator), **Provenance** (builds, sources), and **Policies** (overlays of determinations)
|
||||
* **Edges** like `depends_on`, `contains`, `built_from`, `declared_in`, `affected_by`, `vex_exempts`, `governs_with`
|
||||
* **Time/version** dimension: multiple SBOM snapshots with diffs
|
||||
|
||||
It’s built for investigation and review: find where a vulnerable package enters; see which apps are impacted; understand why a finding exists; simulate a policy version and see the delta. The explorer observes **AOC enforcement**: it never mutates facts; it aggregates and visualizes them. Only the Policy Engine may classify, and classification is displayed as overlays.
|
||||
|
||||
---
|
||||
|
||||
## 2) Why
|
||||
|
||||
* SBOMs are graphs. Tables flatten what matters and hide transitive risk.
|
||||
* Engineers, security, and auditors need impact answers quickly: “What pulls in `log4j:2.17` and where is it at runtime?”
|
||||
* Policy/VEX/advisory interactions are nuanced. A visual overlay makes precedence and outcomes obvious.
|
||||
* Review is collaborative; you need saved queries, deep links, exports, and consistent evidence.
|
||||
|
||||
---
|
||||
|
||||
## 3) How it should work (maximum detail)
|
||||
|
||||
### 3.1 Domain model
|
||||
|
||||
**Nodes** (typed, versioned, tenant‑scoped):
|
||||
|
||||
* `Artifact`: application, service, container image, library, module
|
||||
* `Package`: name + ecosystem (purl), `PackageVersion` with resolved version
|
||||
* `File`: path within artifact or image layer
|
||||
* `License`: SPDX id
|
||||
* `Advisory`: normalized advisory id (GHSA, CVE, vendor), source = Conseiller
|
||||
* `VEX`: statement with product context, status, justification, source = Excitator
|
||||
* `SBOM`: ingestion unit; includes metadata (tool, sha, build info)
|
||||
* `PolicyDetermination`: materialized view of Policy Engine results (read‑only overlay)
|
||||
* `Build`: provenance, commit, workflow run
|
||||
* `Source`: repo, tag, commit
|
||||
|
||||
**Edges** (directed):
|
||||
|
||||
* `declared_in` (PackageVersion → SBOM)
|
||||
* `contains` (Artifact → PackageVersion | File)
|
||||
* `depends_on` (PackageVersion → PackageVersion) with scope attr (prod|dev|test|optional)
|
||||
* `built_from` (Artifact → Build), `provenance_of` (Build → Source)
|
||||
* `affected_by` (PackageVersion → Advisory) with range semantics
|
||||
* `vex_exempts` (Advisory ↔ VEX) scoped by product/component
|
||||
* `licensed_under` (Artifact|PackageVersion → License)
|
||||
* `governs_with` (Artifact|PackageVersion → PolicyDetermination)
|
||||
* `derived_from` (SBOM → SBOM) for superseding snapshots
|
||||
|
||||
**Identity & versioning**
|
||||
|
||||
* Every node has a stable key: `{tenant}:{type}:{natural_id}` (e.g., purl for packages, digest for images).
|
||||
* SBOM snapshots are immutable; edges carry `valid_from`/`valid_to` for time travel and diffing.
|
||||
|
||||
> **Imposed rule:** Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.
|
||||
|
||||
### 3.2 User capabilities (end‑to‑end)
|
||||
|
||||
* **Search & Navigate**: global search (purls, CVEs, repos, licenses), keyboard nav, breadcrumbs, semantic zoom.
|
||||
* **Lenses**: toggle views (Security, License, Provenance, Runtime vs Dev, Policy effect).
|
||||
* **Overlays**:
|
||||
|
||||
* **Advisory overlay**: show affected nodes/edges with source, severity, ranges.
|
||||
* **VEX overlay**: show suppressions/justifications; collapse exempted paths.
|
||||
* **Policy overlay**: choose a policy version; nodes/edges reflect determinations (severity, status) with explain sampling.
|
||||
* **Impact analysis**: pick a vulnerable node; highlight upstream/downstream dependents, scope filters, shortest/all paths with constraints.
|
||||
* **Diff view**: compare SBOM A vs B; show added/removed nodes/edges, changed versions, changed determinations.
|
||||
* **Saved queries**: visual builder + JSON query; shareable permalinks scoped by tenant and environment.
|
||||
* **Exports**: GraphML, CSV edge list, NDJSON of findings, PNG/SVG snapshot.
|
||||
* **Evidence details**: side panel with raw facts, advisory links, VEX statements, policy explain trace, provenance.
|
||||
* **Accessibility**: tab‑navigable, high‑contrast, screen‑reader labels for nodes and sidebars.
|
||||
|
||||
### 3.3 Query model
|
||||
|
||||
* **Visual builder** for common queries:
|
||||
|
||||
* “Show all paths from Artifact X to Advisory Y up to depth 6.”
|
||||
* “All runtime dependencies with license = GPL‑3.0.”
|
||||
* “All artifacts affected by GHSA‑… with no applicable VEX.”
|
||||
* “Which SBOMs introduced/removed `openssl` between build 120 and 130?”
|
||||
* **JSON query** (internal, POST body) with:
|
||||
|
||||
* `start`: list of node selectors (type + id or attributes)
|
||||
* `expand`: edge types and depth, direction, scope filters
|
||||
* `where`: predicates on node/edge attributes
|
||||
* `overlay`: policy version id, advisory sources, VEX filters
|
||||
* `limit`: nodes, edges, timebox, cost budget
|
||||
* **Cost control**: server estimates cost, denies or pages results; UI streams partial graph tiles.
|
||||
|
||||
### 3.4 UI architecture (Console)
|
||||
|
||||
* **Canvas**: WebGL renderer with level‑of‑detail, edge bundling, and label culling; deterministic layout when possible (seeded).
|
||||
* **Semantic zoom**:
|
||||
|
||||
* Far: clusters by artifact/repo/ecosystem, color by lens
|
||||
* Mid: package groups, advisory badges, license swatches
|
||||
* Near: concrete versions, direct edges, inline badges for policy determinations
|
||||
* **Panels**:
|
||||
|
||||
* Left: search, filters, lens selector, saved queries
|
||||
* Right: details, explain trace, evidence tabs (Advisory/VEX/Policy/Provenance)
|
||||
* Bottom: query expression, diagnostics, performance/stream status
|
||||
* **Diff mode**: split or overlay, color legend (add/remove/changed), filter by node type.
|
||||
* **Deep links**: URL encodes query + viewport; shareable respecting RBAC.
|
||||
* **Keyboard**: space drag, +/- zoom, F to focus, G to expand neighbors, P to show paths.
|
||||
|
||||
### 3.5 Back‑end architecture
|
||||
|
||||
**Graph Indexer (new)**
|
||||
|
||||
* Consumes SBOM ingests, Conseiller advisories, Excitator VEX statements, Policy Engine determinations (read‑only).
|
||||
* Projects facts into a **property graph** persisted in:
|
||||
|
||||
* Primary: document store + adjacency sets (e.g., Mongo collections + compressed adjacency lists)
|
||||
* Optional driver for graph DB backends if needed (pluggable)
|
||||
* Maintains materialized aggregates: degree, critical paths cache, affected artifact counts, license distribution.
|
||||
* Emits **graph snapshots** per SBOM with lineage to original ingestion.
|
||||
|
||||
**Graph API (new)**
|
||||
|
||||
* Endpoints for search, neighbor expansion, path queries, diffs, overlays, exports.
|
||||
* Streaming responses for large graphs (chunked NDJSON tiles).
|
||||
* Cost accounting + quotas per tenant.
|
||||
|
||||
**Workers**
|
||||
|
||||
* **Centrality & clustering** precompute on idle: betweenness approximations, connected components, Louvain clusters.
|
||||
* **Diff compute** on new SBOM ingestion pairs (previous vs current).
|
||||
* **Overlay materialization** cache for popular policy versions.
|
||||
|
||||
**Policy Engine integration**
|
||||
|
||||
* Graph API requests can specify a policy version.
|
||||
* For sampled nodes, the API fetches explain traces; for counts, uses precomputed overlay materializations where available.
|
||||
|
||||
**AOC enforcement**
|
||||
|
||||
* Graph Indexer never merges or edits advisories/VEX; it links them and exposes overlays that the Policy Engine evaluates.
|
||||
* Conseiller and Excitator remain authoritative sources; severities come from Policy‑governed normalization.
|
||||
|
||||
### 3.6 APIs (representative)
|
||||
|
||||
* `GET /graph/search?q=...&type=package|artifact|advisory|license`
|
||||
* `POST /graph/query` ⇒ stream tiles `{nodes[], edges[], stats, cursor}`
|
||||
* `POST /graph/paths` body: `{from, to, depth<=6, constraints{scope, runtime_only}}`
|
||||
* `POST /graph/diff` body: `{sbom_a, sbom_b, filters}`
|
||||
* `GET /graph/snapshots/{sbom_id}` ⇒ graph metadata, counts, top advisories
|
||||
* `POST /graph/export` body: `{format: graphml|csv|ndjson|png|svg, query|snapshot}`
|
||||
* `GET /graph/saved` / `POST /graph/saved` save and list tenant queries
|
||||
* `GET /graph/overlays/policy/{version_id}` ⇒ summary stats for caching
|
||||
|
||||
All endpoints tenant‑scoped, RBAC‑checked. Timeouts and pagination by server. Errors return structured diagnostics.
|
||||
|
||||
### 3.7 CLI
|
||||
|
||||
```
|
||||
stella sbom graph search "purl:pkg:maven/org.apache.logging.log4j/log4j-core"
|
||||
stella sbom graph query --file ./query.json --export graphml > graph.graphml
|
||||
stella sbom graph impacted --advisory GHSA-xxxx --runtime-only --limit 100
|
||||
stella sbom graph paths --from artifact:service-a --to advisory:GHSA-xxxx --depth 5 --policy 1.3.0
|
||||
stella sbom graph diff --sbom-a 2025-03-15T10:00Z --sbom-b 2025-03-22T10:00Z --export csv > diff.csv
|
||||
stella sbom graph save --name "openssl-runtime" --file ./query.json
|
||||
```
|
||||
|
||||
Exit codes: 0 ok, 2 query validation error, 3 over‑budget, 4 not found, 5 RBAC denied.
|
||||
|
||||
> **Imposed rule:** Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.
|
||||
|
||||
### 3.8 Performance & scale
|
||||
|
||||
* **Progressive loading**: server pages tiles by BFS frontier; client renders incrementally.
|
||||
* **Viewport culling**: only visible nodes/edges in canvas; offscreen demoted to aggregates.
|
||||
* **Level‑of‑detail**: simplified glyphs and collapsed clusters at distance.
|
||||
* **Query budgets**: per‑tenant rate + node/edge caps; interactive paths limited to depth ≤ 6.
|
||||
* **Caching**: hot queries memoized per tenant + overlay version; diffs precomputed for consecutive SBOMs.
|
||||
|
||||
### 3.9 Security
|
||||
|
||||
* Multi‑tenant isolation at storage and API layers.
|
||||
* RBAC roles:
|
||||
|
||||
* **Viewer**: browse graphs, saved queries
|
||||
* **Investigator**: run queries, export data
|
||||
* **Operator**: configure budgets, purge caches
|
||||
* **Auditor**: download evidence bundles
|
||||
* Input validation for query JSON; deny disallowed edge traversals; strict CSP in web app.
|
||||
|
||||
### 3.10 Observability
|
||||
|
||||
* Metrics: tile latency, nodes/edges per tile, cache hit rate, query denials, memory pressure.
|
||||
* Logs: structured, include query hash, cost, truncation flags.
|
||||
* Traces: server spans per stage (parse, plan, fetch, overlay, stream).
|
||||
|
||||
### 3.11 Accessibility & UX guarantees
|
||||
|
||||
* Keyboard complete, ARIA roles for graph and panels, high‑contrast theme.
|
||||
* Deterministic layout on reload for shareable investigations.
|
||||
|
||||
### 3.12 Data retention
|
||||
|
||||
* Graph nodes derived from SBOMs share retention with SBOM artifacts; overlays are ephemeral caches.
|
||||
* Saved queries retained until deleted; references to missing objects show warnings.
|
||||
|
||||
---
|
||||
|
||||
## 4) Implementation plan
|
||||
|
||||
### 4.1 Services
|
||||
|
||||
* **Graph Indexer (new microservice)**
|
||||
|
||||
* Subscribes to SBOM ingest events, Conseiller advisory updates, Excitator VEX updates, Policy overlay materializations.
|
||||
* Builds adjacency lists and node documents; computes aggregates and clusters.
|
||||
|
||||
* **Graph API (new microservice)**
|
||||
|
||||
* Validates and executes queries; streams tiles; composes overlays; serves diffs and exports.
|
||||
* Integrates with Policy Engine for explain sample retrieval.
|
||||
|
||||
* **SBOM Service (existing)**
|
||||
|
||||
* Emits ingestion events with SBOM ids and lineage; exposes SBOM metadata to Graph API.
|
||||
|
||||
* **Web API Gateway**
|
||||
|
||||
* Routes `/graph/*`, injects tenant context, enforces RBAC.
|
||||
|
||||
### 4.2 Console (Web UI) feature module
|
||||
|
||||
* `packages/features/graph-explorer`
|
||||
|
||||
* Canvas renderer (WebGL), panels, query builder, diff mode, overlays, exports.
|
||||
* Deep‑link router and viewport state serializer.
|
||||
|
||||
### 4.3 Workers
|
||||
|
||||
* Centrality/clustering worker, diff worker, overlay materialization worker.
|
||||
* Schedules on low‑traffic windows; backpressure aware.
|
||||
|
||||
### 4.4 Data model (storage)
|
||||
|
||||
* Collections:
|
||||
|
||||
* `graph_nodes`: `{_id, tenant, type, natural_id, attrs, degree, created_at, updated_at}`
|
||||
* `graph_edges`: `{_id, tenant, from_id, to_id, type, attrs, valid_from, valid_to}`
|
||||
* `graph_snapshots`: per‑SBOM node/edge references
|
||||
* `graph_saved_queries`: `{_id, tenant, name, query_json, created_by}`
|
||||
* `graph_overlays_cache`: keyed by `{tenant, policy_version, hash(query)}`
|
||||
* Indexes: compound on `{tenant, type, natural_id}`, `{tenant, from_id}`, `{tenant, to_id}`, time bounds.
|
||||
|
||||
---
|
||||
|
||||
## 5) Documentation changes (create/update)
|
||||
|
||||
1. **`/docs/sbom/graph-explorer-overview.md`**
|
||||
|
||||
* Concepts, node/edge taxonomy, lenses, overlays, roles, limitations.
|
||||
2. **`/docs/sbom/graph-using-the-console.md`**
|
||||
|
||||
* Walkthroughs: search, navigate, impact, diff, export; screenshots and keyboard cheatsheet.
|
||||
3. **`/docs/sbom/graph-query-language.md`**
|
||||
|
||||
* JSON schema, examples, constraints, cost/budget rules.
|
||||
4. **`/docs/sbom/graph-api.md`**
|
||||
|
||||
* REST endpoints, request/response examples, streaming and pagination.
|
||||
5. **`/docs/sbom/graph-cli.md`**
|
||||
|
||||
* CLI command reference and example pipelines.
|
||||
6. **`/docs/policy/graph-overlays.md`**
|
||||
|
||||
* How policy versions render in Graph; explain sampling; AOC guardrails.
|
||||
7. **`/docs/vex/graph-integration.md`**
|
||||
|
||||
* How VEX suppressions appear and how to validate product scoping.
|
||||
8. **`/docs/advisories/graph-integration.md`**
|
||||
|
||||
* Advisory linkage and severity normalization by policy.
|
||||
9. **`/docs/architecture/graph-services.md`**
|
||||
|
||||
* Graph Indexer, Graph API, storage choices, failure modes.
|
||||
10. **`/docs/observability/graph-telemetry.md`**
|
||||
|
||||
* Metrics, logs, tracing, dashboards.
|
||||
11. **`/docs/runbooks/graph-incidents.md`**
|
||||
|
||||
* Handling runaway queries, cache poisoning, degraded render.
|
||||
12. **`/docs/security/graph-rbac.md`**
|
||||
|
||||
* Permissions matrix, multi‑tenant boundaries.
|
||||
|
||||
Every doc should end with a “Compliance checklist.”
|
||||
**Imposed rule:** Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.
|
||||
|
||||
---
|
||||
|
||||
## 6) Tasks
|
||||
|
||||
### 6.1 Backend: Graph Indexer
|
||||
|
||||
* [ ] Define node/edge schemas and attribute dictionaries for each type.
|
||||
* [ ] Implement event consumers for SBOM ingests, Conseiller updates, Excitator updates.
|
||||
* [ ] Build ingestion pipeline that populates nodes/edges and maintains `valid_from/valid_to`.
|
||||
* [ ] Implement aggregate counters and degree metrics.
|
||||
* [ ] Implement clustering job and persist cluster ids per node.
|
||||
* [ ] Implement snapshot materialization per SBOM and lineage tracking.
|
||||
* [ ] Unit tests for each node/edge builder; property‑based tests for identity stability.
|
||||
|
||||
### 6.2 Backend: Graph API
|
||||
|
||||
* [ ] Implement `/graph/search` with prefix and exact match across node types.
|
||||
* [ ] Implement `/graph/query` with validation, planning, cost estimation, and streaming tile results.
|
||||
* [ ] Implement `/graph/paths` with constraints and depth limits; shortest path heuristic.
|
||||
* [ ] Implement `/graph/diff` computing adds/removes/changed versions; stream results.
|
||||
* [ ] Implement overlays: advisory join, VEX join, policy materialization and explain sampling.
|
||||
* [ ] Implement exports: GraphML, CSV edge list, NDJSON findings, PNG/SVG snapshots.
|
||||
* [ ] RBAC middleware integration; multi‑tenant scoping.
|
||||
* [ ] Load tests with synthetic large SBOMs; define default budgets.
|
||||
|
||||
### 6.3 Policy Engine integration
|
||||
|
||||
* [ ] Add endpoint to fetch explain traces for specific node ids in batch.
|
||||
* [ ] Add materialization export that Graph API can cache per policy version.
|
||||
|
||||
### 6.4 Console (Web UI)
|
||||
|
||||
* [ ] Create `graph-explorer` module with routes `/graph`, `/graph/diff`, `/graph/q/:id`.
|
||||
* [ ] Implement WebGL canvas with LOD, culling, edge bundling, deterministic layout seed.
|
||||
* [ ] Build search, filter, lens, and overlay toolbars.
|
||||
* [ ] Side panels: details, evidence tabs, explain viewer.
|
||||
* [ ] Diff mode: split/overlay toggles and color legend.
|
||||
* [ ] Saved queries: create, update, run; deep links.
|
||||
* [ ] Export UI: formats, server round‑trip, progress indicators.
|
||||
* [ ] a11y audit and keyboard‑only flow.
|
||||
|
||||
### 6.5 CLI
|
||||
|
||||
* [ ] Implement `stella sbom graph *` subcommands with JSON IO and piping support.
|
||||
* [ ] Document examples and stable output schemas for CI consumption.
|
||||
|
||||
### 6.6 Observability & Ops
|
||||
|
||||
* [ ] Dashboards for tile latency, query denials, cache hit rate, memory.
|
||||
* [ ] Alerting on query error spikes, OOM risk, cache churn.
|
||||
* [ ] Runbooks in `/docs/runbooks/graph-incidents.md`.
|
||||
|
||||
### 6.7 Docs
|
||||
|
||||
* [ ] Author all docs in section 5, link from Console contextual help.
|
||||
* [ ] Add end‑to‑end tutorial: “Investigate GHSA‑XXXX across prod artifacts.”
|
||||
|
||||
> **Imposed rule:** Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.
|
||||
|
||||
---
|
||||
|
||||
## 7) Acceptance criteria
|
||||
|
||||
* Console renders large SBOM graphs with semantic zoom, overlays, and responsive interactions.
|
||||
* Users can run impact and path queries with bounded depth and get results within budget.
|
||||
* VEX suppressions and advisory severities appear correctly and are consistent with policy.
|
||||
* Diff view clearly shows added/removed/changed nodes/edges between two SBOMs.
|
||||
* Saved queries and deep links reproduce the same view deterministically (given same data).
|
||||
* Exports produce valid GraphML/CSV/NDJSON and image snapshots.
|
||||
* CLI supports search, query, paths, impacted, diff, and export with stable schemas.
|
||||
* AOC guardrails: explorer never mutates facts; overlays reflect Policy Engine decisions.
|
||||
* RBAC enforced; all actions logged and observable.
|
||||
|
||||
---
|
||||
|
||||
## 8) Risks & mitigations
|
||||
|
||||
* **Graph explosion on large monorepos** → tiling, clustering, budgets, and strict depth limits.
|
||||
* **Inconsistent identities across tools** → canonicalize purls/digests; property‑based tests for identity stability.
|
||||
* **Policy overlay latency** → precompute materializations for hot policy versions; sample explains only on focus.
|
||||
* **User confusion** → strong lens defaults, deterministic layouts, legends, in‑context help.
|
||||
|
||||
---
|
||||
|
||||
## 9) Test plan
|
||||
|
||||
* **Unit**: node/edge builders, identity normalization, cost estimator.
|
||||
* **Integration**: ingest SBOM + advisories + VEX, verify overlays and counts.
|
||||
* **E2E**: Playwright flows for search→impact→diff→export; deep link determinism.
|
||||
* **Performance**: simulate 500k nodes/2M edges; measure tile latency and memory.
|
||||
* **Security**: RBAC matrix; tenant isolation tests; query validation fuzzing.
|
||||
* **Determinism**: snapshot round‑trip: same query and seed produce identical layout and stats.
|
||||
|
||||
---
|
||||
|
||||
## 10) Feature flags
|
||||
|
||||
* `graph.explorer` (UI feature module)
|
||||
* `graph.paths` (advanced path queries)
|
||||
* `graph.diff` (SBOM diff mode)
|
||||
* `graph.overlays.policy` (policy overlay + explain sampling)
|
||||
* `graph.export` (exports enabled)
|
||||
|
||||
Documented in `/docs/observability/graph-telemetry.md`.
|
||||
|
||||
---
|
||||
|
||||
## 11) Non‑goals (this epic)
|
||||
|
||||
* Real‑time process/runtime call graphs.
|
||||
* Full substitution for text reports; Explorer complements Reports.
|
||||
* Cross‑tenant graphs; all queries are tenant‑scoped.
|
||||
|
||||
---
|
||||
|
||||
## 12) Philosophy
|
||||
|
||||
* **See the system**: security and license risk are structural. If you cannot see structure, you will miss risk.
|
||||
* **Evidence over assertion**: every colored node corresponds to raw facts and explainable determinations.
|
||||
* **Bounded interactivity**: fast, partial answers beat slow “complete” ones.
|
||||
* **Immutability**: graphs mirror SBOM snapshots and are never rewritten; we add context, not edits.
|
||||
|
||||
> Final reminder: **Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.**
|
||||
650
docs/implplan/EPIC_6.md
Normal file
650
docs/implplan/EPIC_6.md
Normal file
@@ -0,0 +1,650 @@
|
||||
Below is the expanded, “maximum documentation” package for Epic 6. It is paste‑ready for your repo and deliberately formal so engineering, docs, and audit folks can work from the same source of truth.
|
||||
|
||||
> **Imposed rule:** Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.
|
||||
|
||||
---
|
||||
|
||||
# Epic 6: Vulnerability Explorer (policy‑aware)
|
||||
|
||||
**Short name:** `Vuln Explorer`
|
||||
**Services touched:** Conseiller (Feedser), Excitator (Vexer), SBOM Service, Policy Engine, Findings Ledger (new), Web API Gateway, Authority (authN/Z), Workers/Scheduler, Telemetry/Analytics, Console (Web UI), CLI
|
||||
**AOC ground rule:** Conseiller/Excitator aggregate but never merge or rewrite source documents. The Explorer only renders **effective** results as decided by Policy Engine and records human workflow as immutable ledger events.
|
||||
|
||||
---
|
||||
|
||||
## 1) What it is
|
||||
|
||||
The Vulnerability Explorer is the API and UI for operational triage, investigation, and reporting of vulnerabilities across all artifacts tracked by StellaOps. It correlates SBOM inventory with advisory and VEX evidence, then displays the **effective** status and obligations per the currently selected **Policy version**. It never edits source evidence. It provides:
|
||||
|
||||
* Policy‑aware lists, pivots, and detail views
|
||||
* Triage workflow with immutable audit ledger
|
||||
* Risk acceptance with expiry and evidence
|
||||
* SLA tracking by severity and business tier
|
||||
* Simulation against other policy versions
|
||||
* Exports and cryptographically signed “evidence bundles”
|
||||
|
||||
**Identity principle:** one **Finding** per tuple `(artifact_id, purl, version, advisory_key)`, with links to every contributing AdvisoryEvidence and VexEvidence.
|
||||
|
||||
---
|
||||
|
||||
## 2) Why (concise)
|
||||
|
||||
* Prioritization must reflect **policy and VEX**, not raw feeds.
|
||||
* Audit requires complete, reproducible lineage: what changed, why, and who decided.
|
||||
* Operators need consistent APIs, CLI, and a UI that explain determinations, not just list CVEs.
|
||||
|
||||
---
|
||||
|
||||
## 3) How it should work (maximum detail)
|
||||
|
||||
### 3.1 Domain model and contracts
|
||||
|
||||
**Evidence (immutable)**
|
||||
|
||||
* `AdvisoryEvidence` (from Conseiller)
|
||||
|
||||
```json
|
||||
{
|
||||
"id": "adv_evd:...uuid...",
|
||||
"tenant": "acme",
|
||||
"source": "ghsa|nvd|vendor|ossindex|... ",
|
||||
"source_id": "GHSA-xxxx",
|
||||
"schema": "GHSA|CVE|CSAF-ADV",
|
||||
"advisory_key": "CVE-2024-12345",
|
||||
"affected": [{"ecosystem":"npm","purl":"pkg:npm/lodash","ranges":[{"type":"semver","events":[{"introduced":"0.0.0"},{"fixed":"4.17.21"}]}]}],
|
||||
"cvss": {"version":"3.1","baseScore":7.5,"vectorString":"AV:N/AC:L/..."},
|
||||
"severity": "HIGH",
|
||||
"urls": ["https://..."],
|
||||
"published": "2024-06-10T12:00:00Z",
|
||||
"withdrawn": null,
|
||||
"ingested_at": "2024-06-11T08:43:21Z"
|
||||
}
|
||||
```
|
||||
* `VexEvidence` (from Excitator)
|
||||
|
||||
```json
|
||||
{
|
||||
"id": "vex_evd:...uuid...",
|
||||
"tenant": "acme",
|
||||
"source": "vendor-csaf",
|
||||
"schema": "CSAF-VEX",
|
||||
"advisory_key": "CVE-2024-12345",
|
||||
"product_scope": [{"purl":"pkg:npm/lodash@4.17.20"}],
|
||||
"status": "not_affected|affected|fixed|under_investigation",
|
||||
"justification": "component_not_present|vulnerable_code_not_in_execute_path|... ",
|
||||
"impact_statement": "Not reachable in Acme Payment Service.",
|
||||
"timestamp": "2024-06-12T10:00:00Z",
|
||||
"ingested_at": "2024-06-12T10:10:02Z"
|
||||
}
|
||||
```
|
||||
* `InventoryEvidence` (from SBOM Service)
|
||||
|
||||
```json
|
||||
{
|
||||
"id": "inv_evd:...uuid...",
|
||||
"tenant": "acme",
|
||||
"artifact_id": "svc:payments@1.14.0",
|
||||
"sbom_id": "sbom:sha256:...",
|
||||
"purl": "pkg:npm/lodash@4.17.20",
|
||||
"scope": "runtime|dev|test",
|
||||
"runtime_flag": true,
|
||||
"paths": [["root", "pkg:npm/a", "pkg:npm/lodash"]],
|
||||
"discovered_at": "2024-06-12T11:00:00Z"
|
||||
}
|
||||
```
|
||||
|
||||
**PolicyDetermination** (read‑only from Policy Engine)
|
||||
|
||||
```json
|
||||
{
|
||||
"key": {
|
||||
"artifact_id": "svc:payments@1.14.0",
|
||||
"purl": "pkg:npm/lodash",
|
||||
"version": "4.17.20",
|
||||
"advisory_key": "CVE-2024-12345",
|
||||
"policy_version": "1.3.0"
|
||||
},
|
||||
"applicable": true,
|
||||
"effective_severity": "HIGH",
|
||||
"exploitability": "ACTIVE|LIKELY|UNKNOWN|UNLIKELY",
|
||||
"signals": {"epss": 0.86, "kev": true, "maturity": "weaponized"},
|
||||
"suppression_state": "none|policy|vex",
|
||||
"obligations": [{"type":"fix_by","due":"2024-07-15"},{"type":"document_risk"}],
|
||||
"sla": {"due":"2024-07-15","tier":"gold"},
|
||||
"rationale": [
|
||||
{"rule":"sev.base=nvd","detail":"NVD base:7.5"},
|
||||
{"rule":"exploit.upsell.kev","detail":"KEV flag → HIGH"},
|
||||
{"rule":"env.weight.prod","detail":"Env=prod no downgrade"}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
**Finding identity**
|
||||
|
||||
```
|
||||
finding_id = hash(tenant, artifact_id, purl, version, advisory_key)
|
||||
```
|
||||
|
||||
**Ledger event** (append‑only, tamper‑evident)
|
||||
|
||||
```json
|
||||
{
|
||||
"event_id": "led:...uuid...",
|
||||
"finding_id": "f:...hash...",
|
||||
"tenant": "acme",
|
||||
"type": "assign|comment|attach|change_state|accept_risk|set_target_fix|verify_fix|reopen",
|
||||
"payload": {"to":"team:platform","reason":"oncall triage"},
|
||||
"actor": {"user_id":"u:42","display":"Dana S."},
|
||||
"ts": "2024-06-12T14:01:02Z",
|
||||
"prev_event_hash": "sha256:...",
|
||||
"event_hash": "sha256:sha256(canonical_json(event) + prev_event_hash)"
|
||||
}
|
||||
```
|
||||
|
||||
**Projection** (materialized current state for fast lists)
|
||||
|
||||
```json
|
||||
{
|
||||
"finding_id":"f:...hash...",
|
||||
"tenant":"acme",
|
||||
"artifact_id":"svc:payments@1.14.0",
|
||||
"purl":"pkg:npm/lodash",
|
||||
"version":"4.17.20",
|
||||
"advisory_key":"CVE-2024-12345",
|
||||
"effective_severity":"HIGH",
|
||||
"exploitability":"ACTIVE",
|
||||
"suppression_state":"none",
|
||||
"status":"UNDER_INVESTIGATION|REMEDIATING|ACCEPTED|RESOLVED|NEW|SUPPRESSED",
|
||||
"sla_due":"2024-07-15",
|
||||
"owner":"team:platform",
|
||||
"kev":true,
|
||||
"epss":0.86,
|
||||
"new_since":"2024-06-12",
|
||||
"has_fix":true,
|
||||
"envs":["prod"],
|
||||
"runtime_flag":true,
|
||||
"updated_at":"2024-06-12T14:01:02Z"
|
||||
}
|
||||
```
|
||||
|
||||
**Advisory key normalization**
|
||||
|
||||
* Input identifiers: `CVE-*`, `GHSA-*`, vendor IDs.
|
||||
* Preference order: CVE, then GHSA, else vendor id prefixed with namespace.
|
||||
* Canonicalization: uppercase, trim, map withdrawn to same key but mark `withdrawn=true` in evidence.
|
||||
* Conseiller must publish `links: [all source ids]` for provenance.
|
||||
|
||||
### 3.2 Resolver algorithm (candidate findings)
|
||||
|
||||
**Goal:** produce tuples `(artifact_id, purl, version, advisory_key)` where inventory intersects affected ranges and policy deems the path relevant.
|
||||
|
||||
**Pseudocode**
|
||||
|
||||
```
|
||||
for each artifact sbom S:
|
||||
inv = inventory(S)
|
||||
for each advisory evidence A:
|
||||
for each affected package spec in A.affected:
|
||||
for each inv_item in inv where inv_item.purl package == spec.purl package:
|
||||
if version_in_ranges(inv_item.version, spec.ranges, ecosystem):
|
||||
if policy.path_scope_allows(inv_item.scope, inv_item.runtime_flag, inv_item.paths):
|
||||
yield candidate (artifact_id, inv_item.purl, inv_item.version, A.advisory_key)
|
||||
```
|
||||
|
||||
**Version semantics per ecosystem**
|
||||
|
||||
* npm: semver, pre-release excluded unless explicitly in range.
|
||||
* Maven: Maven version rules, handle `-SNAPSHOT`, use maven‑resolver semantics.
|
||||
* PyPI: PEP 440 versioning.
|
||||
* Go: semver with `+incompatible` handling.
|
||||
* OS packages: RPM/DEB epoch:version‑release ordering.
|
||||
|
||||
**Edge cases**
|
||||
|
||||
* Multiple paths: store shortest path and count.
|
||||
* Dev/test scope: policy may exclude or downgrade.
|
||||
* Withdrawn advisories: keep as evidence; Policy can set severity to `NONE`.
|
||||
|
||||
### 3.3 VEX precedence and scoping
|
||||
|
||||
* If any matching **VEX** says `not_affected` scoped to the artifact product/component per CSAF product tree, set `suppression_state="vex"` and `applicable=false`.
|
||||
* If VEX says `fixed` and inventory version is >= fixed version, mark as **Resolved (verified)** after SBOM recrawl confirms.
|
||||
* If VEX `under_investigation`, no suppression; may add a policy grace period obligation.
|
||||
|
||||
### 3.4 Policy evaluation
|
||||
|
||||
* Inputs: candidate tuple + context (artifact env, business tier, ownership, signals, fix availability).
|
||||
* Determinations: applicability, effective severity, exploitability, obligations, SLA, suppression.
|
||||
* Suppression by policy examples: test‑scope only; path through optional deps; package vendored but not linked at runtime.
|
||||
* Simulation: identical input, alternate `policy_version`; returns determinations without side effects.
|
||||
|
||||
### 3.5 API surface (authoritative)
|
||||
|
||||
**List**
|
||||
|
||||
```
|
||||
GET /vuln/findings?policy=1.3.0&sev=high,critical&group_by=artifact&exploit=kev&env=prod&page=1&page_size=100
|
||||
```
|
||||
|
||||
Response
|
||||
|
||||
```json
|
||||
{
|
||||
"page": 1,
|
||||
"page_size": 100,
|
||||
"total": 740,
|
||||
"group_by": "artifact",
|
||||
"results": [
|
||||
{"group":"svc:payments","counts":{"CRITICAL":3,"HIGH":12,"MEDIUM":8},"sla_breaches":2},
|
||||
...
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
**Query (complex filters)**
|
||||
|
||||
```
|
||||
POST /vuln/findings/query
|
||||
```
|
||||
|
||||
```json
|
||||
{
|
||||
"policy": "1.3.0",
|
||||
"filter": {
|
||||
"severity": [">=MEDIUM"],
|
||||
"exploit": ["kev", "epss>=0.8"],
|
||||
"artifact": ["svc:payments", "svc:checkouts"],
|
||||
"status": ["NEW","UNDER_INVESTIGATION"],
|
||||
"env": ["prod"]
|
||||
},
|
||||
"sort": [{"field":"effective_severity","dir":"desc"},{"field":"epss","dir":"desc"}],
|
||||
"page": {"number":1,"size":200}
|
||||
}
|
||||
```
|
||||
|
||||
**Detail**
|
||||
|
||||
```
|
||||
GET /vuln/findings/{finding_id}?policy=1.3.0
|
||||
```
|
||||
|
||||
Returns projection, evidence links, policy rationale, paths, history summary.
|
||||
|
||||
**Workflow**
|
||||
|
||||
```
|
||||
POST /vuln/findings/{id}/assign { "to": "team:platform" }
|
||||
POST /vuln/findings/{id}/comment { "text": "triage notes..." }
|
||||
POST /vuln/findings/{id}/accept-risk { "until":"2025-06-30","reason":"vendor patch pending","evidence":["url|upload_id"] }
|
||||
POST /vuln/findings/{id}/verify-fix { "sbom_id": "sbom:sha256:..." }
|
||||
POST /vuln/findings/{id}/target-fix { "version": "4.17.21" }
|
||||
```
|
||||
|
||||
**Simulation**
|
||||
|
||||
```
|
||||
POST /vuln/simulate
|
||||
{
|
||||
"policy_from": "1.3.0",
|
||||
"policy_to": "1.4.0",
|
||||
"query": { "severity":[">=MEDIUM"], "env":["prod"] }
|
||||
}
|
||||
```
|
||||
|
||||
Response includes per‑finding delta `{before, after, diff}`.
|
||||
|
||||
**Export**
|
||||
|
||||
```
|
||||
POST /vuln/export { "format":"ndjson","scope":{"query":{...}} }
|
||||
```
|
||||
|
||||
Returns a signed bundle (see §3.10).
|
||||
|
||||
**Errors**
|
||||
|
||||
* `400` validation, `403` RBAC, `404` not found, `409` state conflict (idempotency), `429` rate limited, `5xx` server.
|
||||
|
||||
### 3.6 Console (Web UI)
|
||||
|
||||
**Routes**
|
||||
|
||||
* `/vuln` list with saved views
|
||||
* `/vuln/:id` detail drawer state
|
||||
* `/vuln/simulate/:policyVersion` diff mode
|
||||
|
||||
**State shape (client)**
|
||||
|
||||
```ts
|
||||
interface VulnListState {
|
||||
policyVersion: string;
|
||||
filters: {...};
|
||||
sort: [...];
|
||||
columns: string[];
|
||||
viewId?: string;
|
||||
page: {number: number; size: number};
|
||||
}
|
||||
```
|
||||
|
||||
**UX**
|
||||
|
||||
* Virtualized grid with server paging; column chooser; density toggle.
|
||||
* Quick filters: severity, exploit signals, status, env, owner, fix availability.
|
||||
* Detail tabs: Summary, Evidence (raw docs with provenance), Policy (rationale chain), Paths (deep link to Graph Explorer), Fixes, History.
|
||||
* Simulation bar shows delta chips: `+21 HIGH`, `-9 Suppressed by VEX` etc.
|
||||
* Evidence bundle dialog previews scope and size.
|
||||
* a11y: ARIA roles on grid, keyboard shortcuts: `A` assign, `C` comment, `R` accept risk, `V` verify fix.
|
||||
|
||||
### 3.7 CLI
|
||||
|
||||
Commands
|
||||
|
||||
```
|
||||
stella vuln list --policy 1.3.0 --sev high,critical --group-by artifact --env prod --json
|
||||
stella vuln show --id <finding-id> --policy 1.3.0
|
||||
stella vuln simulate --from 1.3.0 --to 1.4.0 --sev '>=medium' --delta --json
|
||||
stella vuln assign --filter 'advisory:CVE-2024-12345 artifact:payments' --to team:platform
|
||||
stella vuln accept-risk --id <finding-id> --until 2025-06-30 --reason "vendor patch pending" --evidence url:https://ticket/123
|
||||
stella vuln verify-fix --id <finding-id> --sbom <sbom-id>
|
||||
```
|
||||
|
||||
Return codes: `0 ok`, `2 invalid args`, `3 budget exceeded`, `4 not found`, `5 denied`.
|
||||
|
||||
### 3.8 Storage schema (illustrative)
|
||||
|
||||
**Tables**
|
||||
|
||||
```sql
|
||||
-- Evidence
|
||||
CREATE TABLE evidence_advisory (...);
|
||||
CREATE INDEX ea_tenant_key ON evidence_advisory(tenant, advisory_key);
|
||||
CREATE TABLE evidence_vex (...);
|
||||
CREATE INDEX ev_tenant_key ON evidence_vex(tenant, advisory_key);
|
||||
CREATE TABLE evidence_inventory (...);
|
||||
CREATE INDEX ei_artifact_purl ON evidence_inventory(tenant, artifact_id, purl);
|
||||
|
||||
-- Ledger
|
||||
CREATE TABLE findings_ledger_events (
|
||||
event_id uuid PRIMARY KEY,
|
||||
finding_id bytea NOT NULL,
|
||||
tenant text NOT NULL,
|
||||
type text NOT NULL,
|
||||
payload jsonb NOT NULL,
|
||||
actor jsonb NOT NULL,
|
||||
ts timestamptz NOT NULL,
|
||||
prev_event_hash bytea,
|
||||
event_hash bytea NOT NULL
|
||||
);
|
||||
CREATE INDEX fle_find_ts ON findings_ledger_events(tenant, finding_id, ts);
|
||||
|
||||
-- Projection
|
||||
CREATE TABLE findings_projection (
|
||||
finding_id bytea PRIMARY KEY,
|
||||
tenant text NOT NULL,
|
||||
artifact_id text NOT NULL,
|
||||
purl text NOT NULL,
|
||||
version text NOT NULL,
|
||||
advisory_key text NOT NULL,
|
||||
policy_version text NOT NULL,
|
||||
effective_severity text NOT NULL,
|
||||
exploitability text,
|
||||
suppression_state text,
|
||||
status text NOT NULL,
|
||||
sla_due date,
|
||||
owner text,
|
||||
kev boolean,
|
||||
epss double precision,
|
||||
envs text[],
|
||||
runtime_flag boolean,
|
||||
updated_at timestamptz NOT NULL
|
||||
);
|
||||
CREATE INDEX fp_query ON findings_projection(tenant, policy_version, effective_severity, status);
|
||||
```
|
||||
|
||||
**Tamper‑evidence**
|
||||
|
||||
* Ledger events use chained SHA‑256 hashes over canonical JSON + previous hash.
|
||||
* Daily Merkle root of all event hashes is anchored to the audit store (and optionally external timestamping service).
|
||||
|
||||
### 3.9 Performance and scaling
|
||||
|
||||
* P95 list endpoint under 600 ms for 100‑row pages at 5M findings/tenant.
|
||||
* Projections denormalize heavy joins; background projector uses idempotent jobs keyed by `(tenant,finding_id,policy_version)`.
|
||||
* Rate limits per tenant and per API key; backpressure on export jobs; exponential retry for projector.
|
||||
|
||||
### 3.10 Evidence bundle format
|
||||
|
||||
* **Container:** ZIP with `manifest.json`, `findings.ndjson`, `advisory_evidence.ndjson`, `vex_evidence.ndjson`, `inventory_evidence.ndjson`, `policy_version.json`, `ledger_events.ndjson`, `CHECKSUMS`.
|
||||
* **Signing:** Detached signature `bundle.sig` using tenant’s org key (Ed25519).
|
||||
* **Manifest**
|
||||
|
||||
```json
|
||||
{"generated_at":"2024-06-12T15:00:00Z","tenant":"acme","policy_version":"1.3.0","scope":{"query":{...}},"counts":{"findings":421}}
|
||||
```
|
||||
|
||||
### 3.11 Observability
|
||||
|
||||
* Metrics (OpenTelemetry):
|
||||
|
||||
* `vuln_findings_list_latency_ms` (histogram)
|
||||
* `vuln_projection_lag_seconds` (gauge)
|
||||
* `vuln_new_findings_total` (counter)
|
||||
* `vuln_sla_breaches_total` (counter by sev, owner)
|
||||
* `vuln_simulation_latency_ms` (histogram)
|
||||
* Logs: structured JSON with `tenant`, `policy_version`, `query_hash`, `result_count`.
|
||||
* Traces: spans for resolver, policy calls, projection builds, export assembly.
|
||||
* PII: redact comments in logs; store attachments encrypted at rest (KMS).
|
||||
|
||||
### 3.12 Security & RBAC
|
||||
|
||||
**Roles**
|
||||
|
||||
* Viewer: GET list/detail/export read scope.
|
||||
* Investigator: Viewer + workflow actions except risk acceptance.
|
||||
* Operator: Investigator + risk acceptance, verify fix, bulk actions.
|
||||
* Auditor: Viewer + evidence bundles and ledger integrity checks.
|
||||
|
||||
**ABAC**
|
||||
|
||||
* Attribute constraints: by `artifact.owner`, `env`, and `business_tier`.
|
||||
* CSRF protection for Console; all POST require anti‑forgery tokens.
|
||||
* Attachments stored with envelope encryption; signed URLs for limited time access.
|
||||
|
||||
### 3.13 Rollout and migrations
|
||||
|
||||
* Feature flags: `vuln.explorer.ui`, `vuln.explorer.simulation`, `vuln.explorer.bulk_actions`, `vuln.explorer.evidence_bundle`.
|
||||
* Phase 1: dark launch API and projections.
|
||||
* Phase 2: UI read‑only list and detail.
|
||||
* Phase 3: workflow actions and exports.
|
||||
* Data backfill: replay advisory/VEX/SBOM events to seed projections.
|
||||
* Compatibility: maintain projection v1 schema for two releases; migration scripts in `/migrations/vuln/`.
|
||||
|
||||
---
|
||||
|
||||
## 4) Implementation plan
|
||||
|
||||
### 4.1 Services
|
||||
|
||||
* **Findings Ledger (new)**
|
||||
|
||||
* Append‑only event store with projector to `findings_projection`.
|
||||
* Event validation and canonicalization; hashing and Merkle root anchoring.
|
||||
|
||||
* **Vuln Explorer API (new)**
|
||||
|
||||
* Query/filter engine with policy parameterization and grouping.
|
||||
* Simulation endpoint.
|
||||
* Export job orchestrator.
|
||||
|
||||
* **Conseiller / Excitator (updates)**
|
||||
|
||||
* Guarantee canonical `advisory_key` and publish `links[]`.
|
||||
* No merges; maintain raw payload snapshots.
|
||||
|
||||
* **Policy Engine (updates)**
|
||||
|
||||
* Batch evaluation endpoint `POST /policy/eval/batch` with `simulate` support.
|
||||
* Return rationale chain with rule IDs.
|
||||
|
||||
* **SBOM Service (updates)**
|
||||
|
||||
* Publish inventory deltas; include `scope`, `runtime_flag`, `paths`.
|
||||
* Nearest safe version hints.
|
||||
|
||||
* **Workers/Scheduler**
|
||||
|
||||
* Resolver job keyed by `(tenant, artifact_id, sbom_id)`; emits candidate tuples.
|
||||
* Recompute on policy activation and evidence changes.
|
||||
|
||||
### 4.2 Code structure
|
||||
|
||||
```
|
||||
/src/Findings/StellaOps.Findings.Ledger
|
||||
/api
|
||||
/projector
|
||||
/storage
|
||||
/src/VulnExplorer/StellaOps.VulnExplorer.Api
|
||||
/routes
|
||||
/query
|
||||
/simulation
|
||||
/export
|
||||
/packages/console/features/vuln-explorer
|
||||
/components
|
||||
/pages
|
||||
/state
|
||||
/src/Cli/StellaOps.Cli
|
||||
```
|
||||
|
||||
### 4.3 Performance tasks
|
||||
|
||||
* Projection indexes and covering queries; explain plans in `/docs/vuln/perf-notes.md`.
|
||||
* Cache hot groupings per tenant with TTL and invalidation on ledger projector tick.
|
||||
|
||||
---
|
||||
|
||||
## 5) Documentation changes (create/update)
|
||||
|
||||
1. `/docs/vuln/explorer-overview.md`
|
||||
Conceptual model, identities, evidence vs determinations, AOC guarantees.
|
||||
2. `/docs/vuln/explorer-using-console.md`
|
||||
Workflows with screenshots, keyboard shortcuts, saved views, deep links.
|
||||
3. `/docs/vuln/explorer-api.md`
|
||||
Endpoint specs, query language, grouping, pagination, errors, rate limits.
|
||||
4. `/docs/vuln/explorer-cli.md`
|
||||
Commands, flags, examples, exit codes.
|
||||
5. `/docs/vuln/findings-ledger.md`
|
||||
Event schema, state machine, hashing, Merkle roots, integrity checks.
|
||||
6. `/docs/policy/vuln-determinations.md`
|
||||
Inputs, outputs, precedence rules, simulation semantics.
|
||||
7. `/docs/vex/explorer-integration.md`
|
||||
CSAF mapping, scoping to product tree, precedence.
|
||||
8. `/docs/advisories/explorer-integration.md`
|
||||
Advisory key normalization, provenance, withdrawn handling.
|
||||
9. `/docs/sbom/vuln-resolution.md`
|
||||
Ecosystem version semantics, path sensitivity, scope rules.
|
||||
10. `/docs/observability/vuln-telemetry.md`
|
||||
Metrics, logs, traces, dashboards, SLOs.
|
||||
11. `/docs/security/vuln-rbac.md`
|
||||
Role mapping, ABAC, attachment encryption, CSRF.
|
||||
12. `/docs/runbooks/vuln-ops.md`
|
||||
Recompute storms, projector lag, policy activation drains, export failures.
|
||||
13. `/docs/install/containers.md`
|
||||
Add `findings-ledger`, `vuln-explorer-api` images, compose/k8s manifests, resource sizing, health checks.
|
||||
|
||||
> Each doc ends with: **Imposed rule:** Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.
|
||||
|
||||
---
|
||||
|
||||
## 6) Engineering tasks
|
||||
|
||||
### Backend: Findings & API
|
||||
|
||||
* [ ] Define evidence and ledger schemas; migrations.
|
||||
* [ ] Implement resolver for npm, Maven, PyPI, Go, OS packages with property‑based tests for version comparisons.
|
||||
* [ ] Implement ledger API and projector with idempotency and hashing.
|
||||
* [ ] Implement list/detail/grouping endpoints with server‑side paging.
|
||||
* [ ] Implement simulation and export (bundle assembly, signing).
|
||||
* [ ] Integrate Policy Engine batch eval with rationale traces.
|
||||
* [ ] RBAC via Authority with ABAC filters.
|
||||
* [ ] Load tests at 5M findings/tenant; tune indexes.
|
||||
|
||||
### Conseiller/Excitator
|
||||
|
||||
* [ ] Normalize `advisory_key` and persist `links[]`.
|
||||
* [ ] Ensure raw payload snapshots are retrievable by Explorer for Evidence tab.
|
||||
|
||||
### SBOM Service
|
||||
|
||||
* [ ] Emit `scope`, `runtime_flag`, `paths`; safe version hints.
|
||||
* [ ] Inventory delta events to trigger resolver.
|
||||
|
||||
### Console
|
||||
|
||||
* [ ] Build grid with virtualization, saved views, deep link serializer.
|
||||
* [ ] Implement detail tabs and path deep‑links to Graph Explorer.
|
||||
* [ ] Add simulation bar and delta chips.
|
||||
* [ ] Evidence bundle dialog.
|
||||
* [ ] a11y keyboard flow and ARIA labeling; unit and E2E tests.
|
||||
|
||||
### CLI
|
||||
|
||||
* [ ] `stella vuln list|show|simulate|assign|accept-risk|verify-fix` with `--json` and CSV export.
|
||||
* [ ] Stable output schemas; pipe‑friendly defaults.
|
||||
|
||||
### Observability/Ops
|
||||
|
||||
* [ ] Dashboards for list latency, projection lag, new/reopened, SLA breaches.
|
||||
* [ ] Alerts on projector backlog, API 5xx spikes, export failures.
|
||||
* [ ] Runbooks in `/docs/runbooks/vuln-ops.md`.
|
||||
|
||||
### Docs
|
||||
|
||||
* [ ] Author files listed in §5 with cross‑links to Policy Studio and SBOM Graph Explorer.
|
||||
* [ ] Update `/docs/install/containers.md` with new images and compose/k8s snippets.
|
||||
|
||||
> **Imposed rule:** Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.
|
||||
|
||||
---
|
||||
|
||||
## 7) Acceptance criteria
|
||||
|
||||
* List and detail views reflect **effective** policy outcomes and update instantly when switching policy versions.
|
||||
* Evidence tab shows all raw advisory/VEX documents with provenance; no source merging.
|
||||
* Resolver respects ecosystem semantics, scope, and paths; path tab round‑trips to Graph Explorer.
|
||||
* Ledger events are immutable and reconstruct historical list states accurately.
|
||||
* Simulation returns diffs without side effects and matches Policy Engine outputs.
|
||||
* CLI/API support paging, grouping, export, simulation; contracts stable and documented.
|
||||
* RBAC and tenant isolation validated by tests; attachments encrypted.
|
||||
* P95 performance budgets met; dashboards green for SLOs.
|
||||
|
||||
---
|
||||
|
||||
## 8) Risks and mitigations
|
||||
|
||||
* **Advisory identity collisions** → strict canonicalization; preserve `links[]`; never merge raw docs.
|
||||
* **Projection lag** → backpressure, worker autoscaling, health checks; alerting on lag.
|
||||
* **Resolver false positives** → path evidence required; dev/test scope rules explicit; ecosystem‑specific tests.
|
||||
* **User confusion over suppression** → explicit badges; Policy tab with rationale and “why changed.”
|
||||
* **Export size** → NDJSON streaming, size estimator in UI, scope previews.
|
||||
|
||||
---
|
||||
|
||||
## 9) Test plan
|
||||
|
||||
* **Unit:** version comparators, resolver per ecosystem, policy mapping, ledger state machine.
|
||||
* **Integration:** SBOM + advisories + VEX ingestion, candidate generation, policy application, suppression precedence.
|
||||
* **E2E Console:** triage, bulk assign, simulation, evidence bundle download; keyboard‑only flow.
|
||||
* **Performance:** list/grouping at target scale; projector rebuild; export assembly.
|
||||
* **Security:** RBAC matrix, ABAC filters, CSRF, signed URL lifetimes, tamper‑evidence verification.
|
||||
* **Determinism:** time‑travel snapshots reproduce prior states byte‑for‑byte.
|
||||
|
||||
---
|
||||
|
||||
## 10) Philosophy
|
||||
|
||||
* **Facts first, decisions second.** Evidence is immutable; decisions and workflow sit on top in a ledger.
|
||||
* **Policy is the lens.** The same facts can imply different obligations; the system must make that explicit and reproducible.
|
||||
* **Audit > convenience.** Every state change is justified, signed, and verifiable.
|
||||
* **No hidden magic.** If anything is suppressed, the UI shows the rule or VEX that did it, with documents attached.
|
||||
|
||||
> Final reminder: **Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.**
|
||||
545
docs/implplan/EPIC_7.md
Normal file
545
docs/implplan/EPIC_7.md
Normal file
@@ -0,0 +1,545 @@
|
||||
Below is the expanded, “maximum documentation” package for Epic 7. It is paste‑ready for your repo and deliberately formal so engineering, docs, governance, and audit can work from the same source of truth.
|
||||
|
||||
> **Imposed rule:** Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.
|
||||
|
||||
---
|
||||
|
||||
# Epic 7: VEX Consensus Lens
|
||||
|
||||
**Short name:** `VEX Lens`
|
||||
**Services touched:** Excitator (Vexer), Conseiller (Feedser), SBOM Service, Policy Engine, Findings Ledger, Web API Gateway, Authority (authN/Z), Console (Web UI), CLI, Telemetry/Analytics
|
||||
**AOC ground rule:** Excitator and Conseiller aggregate but never merge or rewrite source documents. The Lens only computes a **derived, reproducible** consensus view while preserving all raw evidence and provenance.
|
||||
|
||||
---
|
||||
|
||||
## 1) What it is
|
||||
|
||||
The VEX Consensus Lens is a deterministic computation and presentation layer that ingests all available VEX statements about a given advisory key (e.g., CVE) and product/context, then produces an **explicit consensus state** for each `(artifact_id, purl, version, advisory_key)` tuple. It answers: “Given multiple, possibly conflicting VEX statements from different issuers, what is the most trustworthy, policy‑consistent interpretation for this artifact and version?”
|
||||
|
||||
It never edits or merges VEX documents. It normalizes them, aligns their **product scope** to SBOM inventory, scores issuers via a tenant‑configurable **trust model**, applies time and version scoping, and outputs:
|
||||
|
||||
* `consensus_state`: `NOT_AFFECTED | AFFECTED | FIXED | UNDER_INVESTIGATION | INCONCLUSIVE | DISPUTED`
|
||||
* `confidence`: 0.0–1.0 numeric score
|
||||
* `rationale`: structured explanation of evidence, weights, and rules applied
|
||||
* `quorum`: weighted vote breakdown by issuer, timestamp, justification, and version applicability
|
||||
|
||||
The Lens plugs into the Policy Engine. Policy remains the source of truth for applicability, severity, and obligations; the Lens supplies structured signals and provenance that policy can use to suppress findings, downgrade risk, or set SLAs.
|
||||
|
||||
Key properties:
|
||||
|
||||
* **Reproducible:** Same inputs and policy yield same outputs.
|
||||
* **Explainable:** Every output carries a machine‑readable rationale chain.
|
||||
* **Scoped:** Product trees and version ranges are resolved against actual SBOM inventory.
|
||||
* **Immutable evidence:** All raw VEX docs remain intact, retrievable, and linkable.
|
||||
|
||||
---
|
||||
|
||||
## 2) Why (concise)
|
||||
|
||||
Real ecosystems have overlapping VEX statements from upstream maintainers, vendors, distros, and third parties. Operators need a single, defensible view without losing provenance. The Lens reduces noise, flags conflicts, and turns fragmented VEX evidence into auditable signals that policy and humans can act on.
|
||||
|
||||
---
|
||||
|
||||
## 3) How it should work (maximum detail)
|
||||
|
||||
### 3.1 Input model
|
||||
|
||||
**From Excitator (Vexer):**
|
||||
|
||||
* Raw `VexEvidence` documents (CSAF VEX, OpenVEX, CycloneDX VEX) with:
|
||||
|
||||
* `advisory_key` (canonicalized)
|
||||
* product tree or component coordinates (purl, CPE, vendor IDs)
|
||||
* status (`not_affected`, `affected`, `fixed`, `under_investigation`)
|
||||
* justifications (CSAF, OpenVEX enumerations)
|
||||
* version ranges or fixed versions if available
|
||||
* timestamps (issued, last_updated), source URLs
|
||||
* cryptographic metadata (signature, issuer, certificate chain) if present
|
||||
|
||||
**From SBOM Service:**
|
||||
|
||||
* Inventory for each artifact: purls, versions, dependency paths, scopes, env flags.
|
||||
|
||||
**From Issuer Directory (new)**
|
||||
|
||||
* Directory of known VEX issuers:
|
||||
|
||||
* identity, organization, domain, CSAF publisher metadata
|
||||
* public keys and trust anchors (Ed25519/X.509/PKIX/PKCS7/DSSE)
|
||||
* default trust weight
|
||||
* tenancy‑specific overrides
|
||||
|
||||
**From Policy Engine:**
|
||||
|
||||
* Policy version and options relevant to VEX evaluation:
|
||||
|
||||
* trust model parameters
|
||||
* confidence thresholds per environment/tier
|
||||
* justification whitelist/blacklist
|
||||
* recency requirements and expiry windows
|
||||
* precedence rules for status conflicts
|
||||
|
||||
### 3.2 Normalization
|
||||
|
||||
1. **Canonical advisory key:** CVE preferred; GHSA mapped; vendor IDs namespaced.
|
||||
2. **Status mapping:** Normalize all encodings to `NOT_AFFECTED | AFFECTED | FIXED | UNDER_INVESTIGATION`.
|
||||
3. **Product scope alignment:** Convert CSAF product tree/CPE to purl variants using deterministic mappings; store mapping evidence.
|
||||
4. **Version scoping:** For each evidence, compute `applies_to(version)` using ecosystem comparators (npm semver, Maven, PEP 440, Go semver, RPM/DEB EVR).
|
||||
5. **Signature verification:** If signed, verify against Issuer Directory trust anchors; attach `sig_verified=true/false` and chain metadata.
|
||||
6. **Temporal fields:** Compute `effective_at` (issued), `observed_at` (ingested), and staleness.
|
||||
|
||||
### 3.3 Trust model
|
||||
|
||||
Each `VexEvidence` gets a base weight `w_base` from issuer type and verification:
|
||||
|
||||
* Maintainer/Upstream signed: 0.9
|
||||
* Maintainer/Upstream unsigned: 0.7
|
||||
* Distro/SIG signed: 0.8
|
||||
* Vendor of downstream product signed: 0.8
|
||||
* Third‑party scanner VEX: 0.4
|
||||
* Unknown/unsigned with weak provenance: 0.2
|
||||
|
||||
Weights are then adjusted:
|
||||
|
||||
```
|
||||
w = w_base
|
||||
* f_signature(sig_verified) // e.g., 1.1 if verified, 0.8 if unverifiable
|
||||
* f_recency(age_days) // decay after T days (policy)
|
||||
* f_justification(type) // e.g., "component_not_present" lower if SBOM shows present
|
||||
* f_scope_match(score) // quality of product match: exact purl > family > CPE wildcard
|
||||
* f_env(app_env) // optional env-specific multipliers
|
||||
```
|
||||
|
||||
All `f_*` are bounded to [0.5, 1.2] by default to prevent runaway effects. Tenants can override per Policy Studio.
|
||||
|
||||
### 3.4 Consensus algorithm
|
||||
|
||||
For a tuple `(artifact_id, purl, version, advisory_key)`:
|
||||
|
||||
1. **Collect** all normalized `VexEvidence` whose product scope maps to `purl` and `applies_to(version)==true`.
|
||||
|
||||
2. **Bucket** by normalized status; compute `W(status) = Σ w(evidence)` per status.
|
||||
|
||||
3. **Apply precedence** rules from policy:
|
||||
|
||||
* If `W(NOT_AFFECTED)` exceeds threshold `T_na` and there is no `FIXED` evidence contradictory for the same version, propose `NOT_AFFECTED`.
|
||||
* If any `FIXED` applies and inventory version ≥ fixed version, propose `FIXED`.
|
||||
* If `W(AFFECTED)` ≥ `T_aff` and no dominating `NOT_AFFECTED`, propose `AFFECTED`.
|
||||
* If `W(UNDER_INVESTIGATION)` dominates and others below thresholds, propose `UNDER_INVESTIGATION`.
|
||||
* If both `AFFECTED` and `NOT_AFFECTED` exceed thresholds within a small margin `δ`, mark `DISPUTED`.
|
||||
* Otherwise `INCONCLUSIVE`.
|
||||
|
||||
4. **Confidence score:**
|
||||
|
||||
```
|
||||
confidence = W(winning_status) / (W(AFFECTED)+W(NOT_AFFECTED)+W(FIXED)+W(UNDER_INVESTIGATION) + ε)
|
||||
```
|
||||
|
||||
Clip to [0.0, 1.0].
|
||||
|
||||
5. **Rationale chain:**
|
||||
|
||||
* Winning status, thresholds used, top 5 contributing evidences with weights and reasons, product mapping quality, version scoping evidence, and policy knobs that influenced the result.
|
||||
|
||||
6. **Quorum summary:**
|
||||
|
||||
* List issuers and their votes, signature state, timestamps, and justifications.
|
||||
|
||||
### 3.5 Policy interaction
|
||||
|
||||
* The Lens returns `consensus_state`, `confidence`, and structured `signals`.
|
||||
* Policy Engine consumes these and may:
|
||||
|
||||
* Suppress findings automatically on `NOT_AFFECTED` confidence ≥ `P_na`.
|
||||
* Downgrade severity or extend SLA when `UNDER_INVESTIGATION` from high‑trust issuers.
|
||||
* Require human approval when `DISPUTED`.
|
||||
* Treat `FIXED` as resolved only when SBOM crawl verifies the fixed version or a `verify_fix` ledger event exists.
|
||||
|
||||
**Simulation:** Policy Studio can simulate different trust weights and thresholds and see consensus deltas without side effects.
|
||||
|
||||
### 3.6 Data contracts
|
||||
|
||||
**ConsensusRecord (materialized)**
|
||||
|
||||
```json
|
||||
{
|
||||
"id": "cons:sha256(tenant|artifact|purl|version|advisory|policy)",
|
||||
"tenant": "acme",
|
||||
"artifact_id": "svc:payments@1.14.0",
|
||||
"purl": "pkg:npm/lodash@4.17.20",
|
||||
"advisory_key": "CVE-2024-12345",
|
||||
"policy_version": "1.3.0",
|
||||
"consensus_state": "NOT_AFFECTED",
|
||||
"confidence": 0.87,
|
||||
"weights": {"AFFECTED":0.31,"NOT_AFFECTED":1.25,"FIXED":0.00,"UNDER_INVESTIGATION":0.12},
|
||||
"top_evidence": ["vex_evd:...","vex_evd:..."],
|
||||
"quorum": [
|
||||
{"issuer":"lodash-maintainers","status":"NOT_AFFECTED","w":0.9,"sig_verified":true,"just":"vulnerable_code_not_present","issued":"2024-06-08"},
|
||||
{"issuer":"vendorX-distro","status":"AFFECTED","w":0.25,"sig_verified":false,"just":"generic_advisory","issued":"2024-06-07"}
|
||||
],
|
||||
"rationale": [
|
||||
{"rule":"trust.weight.issuer","detail":"maintainer signed evidence 0.9"},
|
||||
{"rule":"scope.match.exact","detail":"exact purl match"},
|
||||
{"rule":"justification.vcnp","detail":"supported by SBOM callgraph hint"}
|
||||
],
|
||||
"updated_at": "2024-06-12T10:00:00Z"
|
||||
}
|
||||
```
|
||||
|
||||
**Issuer Directory entry**
|
||||
|
||||
```json
|
||||
{
|
||||
"issuer_id": "iss:lodash",
|
||||
"org": "Lodash Maintainers",
|
||||
"domains": ["lodash.com"],
|
||||
"keys": [{"type":"ed25519","pub":"...","expires":"2026-12-31"}],
|
||||
"default_weight": 0.9,
|
||||
"metadata": {"csaf_publisher": true}
|
||||
}
|
||||
```
|
||||
|
||||
### 3.7 APIs
|
||||
|
||||
**Compute/Query**
|
||||
|
||||
```
|
||||
GET /vex/consensus?artifact=svc:payments@1.14.0&purl=pkg:npm/lodash@4.17.20&advisory=CVE-2024-12345&policy=1.3.0
|
||||
POST /vex/consensus/query { "policy":"1.3.0", "filter": { "state":["DISPUTED","INCONCLUSIVE"], "confidence":"<0.6", "env":["prod"] }, "page":{...} }
|
||||
GET /vex/consensus/{id} // full record
|
||||
POST /vex/consensus/simulate // override trust knobs and thresholds for what-if
|
||||
```
|
||||
|
||||
**Issuer Directory**
|
||||
|
||||
```
|
||||
GET /vex/issuers
|
||||
POST /vex/issuers (admin)
|
||||
POST /vex/issuers/{id}/keys (admin)
|
||||
```
|
||||
|
||||
**Exports**
|
||||
|
||||
```
|
||||
POST /vex/consensus/export { "format":"ndjson","scope":{"filter":{...}} }
|
||||
```
|
||||
|
||||
**Errors**
|
||||
|
||||
* `400` invalid mapping, `403` RBAC, `404` not found, `409` conflict, `429` rate limit.
|
||||
|
||||
### 3.8 Console (Web UI)
|
||||
|
||||
Routes:
|
||||
|
||||
* `/vex/consensus` overview with filters: state, confidence, issuer, advisory, artifact, env.
|
||||
* `/vex/consensus/:id` detail: Evidence pane, Quorum graph, Policy impact, SBOM path links.
|
||||
|
||||
UX elements:
|
||||
|
||||
* **Quorum bar:** stacked bar showing weights per status; hover reveals issuer contributions.
|
||||
* **Confidence chip:** numeric and qualitative band (Low/Med/High).
|
||||
* **Evidence table:** paged list of VEX docs with signature icon, scope match quality tag, justification tag, issued/updated timestamps.
|
||||
* **Conflict view:** for `DISPUTED`, show side-by-side issuer rationales and suggested next steps.
|
||||
* **Deep link** into Vulnerability Explorer detail, preselecting the Policy version used for the consensus.
|
||||
|
||||
A11y:
|
||||
|
||||
* ARIA roles on grid and bars; keyboard shortcuts `S` switch policy, `T` trust presets, `E` export.
|
||||
|
||||
### 3.9 CLI
|
||||
|
||||
Commands:
|
||||
|
||||
```
|
||||
stella vex consensus list --policy 1.3.0 --state disputed,inconclusive --confidence '<0.6' --artifact payments --json
|
||||
stella vex consensus show --id <cons-id> --policy 1.3.0
|
||||
stella vex simulate --policy 1.3.0 --trust 'issuer:lodash=1.0,issuer:vendorX=0.5' --thresholds 'na=1.0,aff=0.6' --json
|
||||
stella vex issuers list
|
||||
stella vex export --filter 'artifact:payments advisory:CVE-2024-12345' --out vex-consensus.ndjson
|
||||
```
|
||||
|
||||
Exit codes: `0` ok, `2` invalid args, `4` not found, `5` denied.
|
||||
|
||||
### 3.10 Storage schema (illustrative)
|
||||
|
||||
```sql
|
||||
-- Normalized VEX (reference Excitator id; do not alter raw)
|
||||
CREATE TABLE vex_normalized (
|
||||
id uuid PRIMARY KEY,
|
||||
tenant text NOT NULL,
|
||||
evidence_id text NOT NULL, -- link to Excitator
|
||||
advisory_key text NOT NULL,
|
||||
issuer_id text,
|
||||
status text NOT NULL, -- NOT_AFFECTED|AFFECTED|FIXED|UNDER_INVESTIGATION
|
||||
justification text,
|
||||
purl text, -- normalized mapping target
|
||||
version_range jsonb, -- ecosystem-specific encoding
|
||||
sig_verified boolean,
|
||||
scope_score real, -- 0..1 quality of mapping
|
||||
issued timestamptz,
|
||||
updated timestamptz,
|
||||
w_base real,
|
||||
UNIQUE (tenant, evidence_id)
|
||||
);
|
||||
|
||||
-- Issuer Directory
|
||||
CREATE TABLE vex_issuers (
|
||||
issuer_id text PRIMARY KEY,
|
||||
tenant text NOT NULL,
|
||||
org text NOT NULL,
|
||||
default_weight real NOT NULL,
|
||||
metadata jsonb,
|
||||
UNIQUE(tenant, org)
|
||||
);
|
||||
|
||||
CREATE TABLE vex_issuer_keys (
|
||||
id uuid PRIMARY KEY,
|
||||
issuer_id text NOT NULL REFERENCES vex_issuers(issuer_id),
|
||||
key_type text NOT NULL,
|
||||
pubkey text NOT NULL,
|
||||
expires timestamptz
|
||||
);
|
||||
|
||||
-- Consensus projection
|
||||
CREATE TABLE vex_consensus (
|
||||
id bytea PRIMARY KEY,
|
||||
tenant text NOT NULL,
|
||||
artifact_id text NOT NULL,
|
||||
purl text NOT NULL,
|
||||
version text NOT NULL,
|
||||
advisory_key text NOT NULL,
|
||||
policy_version text NOT NULL,
|
||||
consensus_state text NOT NULL,
|
||||
confidence real NOT NULL,
|
||||
weights jsonb NOT NULL,
|
||||
top_evidence text[] NOT NULL,
|
||||
updated_at timestamptz NOT NULL
|
||||
);
|
||||
|
||||
CREATE INDEX vc_query ON vex_consensus(tenant, policy_version, consensus_state, confidence);
|
||||
```
|
||||
|
||||
### 3.11 Integration with Findings Ledger and Vuln Explorer
|
||||
|
||||
* The Vuln Explorer reads `vex_consensus` for each finding and renders:
|
||||
|
||||
* Consensus chip, confidence, and a link to full quorum.
|
||||
* For `NOT_AFFECTED` with confidence ≥ policy threshold, show “Suppressed by VEX (Consensus)” badge.
|
||||
* For `DISPUTED`, open a triage banner prompting manual review and optional ledger comment/assignment.
|
||||
|
||||
* Ledger receives no new event type from lens computation itself. Human actions triggered by lens views produce standard events (`comment`, `assign`, `change_state`).
|
||||
|
||||
### 3.12 Security & RBAC
|
||||
|
||||
Roles:
|
||||
|
||||
* Viewer: query consensus read‑only.
|
||||
* Investigator: Viewer + export.
|
||||
* Operator: Investigator + trust simulation.
|
||||
* Admin: manage Issuer Directory entries and keys.
|
||||
|
||||
CSRF for Console; ABAC scoping by artifact ownership and environment.
|
||||
|
||||
### 3.13 Observability
|
||||
|
||||
Metrics:
|
||||
|
||||
* `vex_consensus_compute_latency_ms` (histogram)
|
||||
* `vex_consensus_records_total` (counter)
|
||||
* `vex_consensus_disputed_total` (counter by issuer combinations)
|
||||
* `vex_consensus_staleness_seconds` (gauge)
|
||||
* `vex_signature_verification_rate` (gauge)
|
||||
|
||||
Logs: structured events with `tenant`, `policy_version`, `advisory_key`, `quorum_summary`.
|
||||
Traces: spans for normalization, mapping, trust weighting, consensus decision, DB writes.
|
||||
|
||||
### 3.14 Performance & scaling
|
||||
|
||||
Targets:
|
||||
|
||||
* P95 query under 500 ms for 100‑row pages at 10M consensus records/tenant.
|
||||
* Projection jobs are idempotent and keyed by `(tenant, artifact, purl, version, advisory, policy)`; backpressure with work queues.
|
||||
* Cache popular queries with tenant‑scoped TTL; invalidate on Excitator or policy changes.
|
||||
|
||||
### 3.15 Edge cases
|
||||
|
||||
* **Ambiguous product mapping:** mark low `scope_score`, cap weight, surface warning in UI.
|
||||
* **VEX “not present” vs SBOM shows present:** down‑weight with `f_justification`, require manual check.
|
||||
* **Withdrawn or superseded VEX:** decay to near zero; keep provenance.
|
||||
* **Partial fixes:** if fixed version applies to subset of platforms, map to env or arch dimension when available.
|
||||
* **Time travel:** consensus recalculated as of a timestamp using only evidence ≤ `as_of` and the corresponding policy version.
|
||||
|
||||
---
|
||||
|
||||
## 4) Implementation plan
|
||||
|
||||
### 4.1 Services
|
||||
|
||||
* **VEX Lens Service (new)**
|
||||
|
||||
* Normalization pipeline, trust weighting, consensus computation, and projections.
|
||||
* Batch recompute on policy activation and Excitator deltas.
|
||||
|
||||
* **Excitator (updates)**
|
||||
|
||||
* Ensure all VEX evidence carries issuer hints and raw signature blobs when present.
|
||||
* Publish product trees and original coordinates intact.
|
||||
|
||||
* **Policy Engine (updates)**
|
||||
|
||||
* Add VEX trust knobs, thresholds, recency decay, and status precedence.
|
||||
* Batch eval endpoint accepts `consensus inputs` where needed.
|
||||
|
||||
* **Issuer Directory (new)**
|
||||
|
||||
* Manage issuer metadata and keys; tenant overrides; audit logs.
|
||||
|
||||
### 4.2 Code structure
|
||||
|
||||
```
|
||||
/src/VexLens/StellaOps.VexLens
|
||||
/normalizer
|
||||
/mapping # CPE/purl translators
|
||||
/trust # weighting functions
|
||||
/consensus # algorithm and projections
|
||||
/api
|
||||
/src/Excititor # updates
|
||||
/src/Policy/__Libraries/StellaOps.Policy # updates
|
||||
/src/IssuerDirectory/StellaOps.IssuerDirectory
|
||||
/packages/console/features/vex-consensus
|
||||
/src/Cli/StellaOps.Cli
|
||||
```
|
||||
|
||||
### 4.3 Rollout
|
||||
|
||||
* Phase 1: API read‑only with basic trust model, Console list/detail, no simulation.
|
||||
* Phase 2: Policy Studio integrations and simulation.
|
||||
* Phase 3: Issuer Directory admin flows, exports, and advanced mapping diagnostics.
|
||||
|
||||
---
|
||||
|
||||
## 5) Documentation changes (create/update)
|
||||
|
||||
1. `/docs/vex/consensus-overview.md`
|
||||
Purpose, scope, terminology, evidence vs derived view, AOC guarantees.
|
||||
|
||||
2. `/docs/vex/consensus-algorithm.md`
|
||||
Normalization, mapping, weighting, thresholds, precedence, formulas, examples.
|
||||
|
||||
3. `/docs/vex/issuer-directory.md`
|
||||
Managing issuers, keys, trust overrides, security model.
|
||||
|
||||
4. `/docs/vex/consensus-api.md`
|
||||
Endpoints, request/response schemas, errors, pagination, rate limits.
|
||||
|
||||
5. `/docs/vex/consensus-console.md`
|
||||
Screens, filters, conflict workflows, a11y, deep links.
|
||||
|
||||
6. `/docs/policy/vex-trust-model.md`
|
||||
Policy knobs, thresholds, decay, simulation.
|
||||
|
||||
7. `/docs/sbom/vex-mapping.md`
|
||||
Product tree mapping to purl/version, ecosystem comparators, edge cases.
|
||||
|
||||
8. `/docs/security/vex-signatures.md`
|
||||
Signature verification flows, key management, auditing.
|
||||
|
||||
9. `/docs/runbooks/vex-ops.md`
|
||||
Recompute storms, mapping failures, signature errors, lag, quotas.
|
||||
|
||||
All docs end with the imposed rule statement.
|
||||
|
||||
---
|
||||
|
||||
## 6) Engineering tasks
|
||||
|
||||
### Backend core
|
||||
|
||||
* [ ] Implement normalization for CSAF VEX, OpenVEX, CycloneDX VEX.
|
||||
* [ ] Build product mapping library (CPE→purl, vendor tokens→purl families).
|
||||
* [ ] Implement signature verification (Ed25519/PKIX/DSSE) using Issuer Directory keys.
|
||||
* [ ] Implement trust weighting functions and configurable parameters.
|
||||
* [ ] Implement consensus algorithm with unit tests and property tests.
|
||||
* [ ] Materialize `vex_consensus` projection with indexes and idempotent workers.
|
||||
* [ ] Batch recompute on policy activation and Excitator deltas.
|
||||
|
||||
### APIs & Integrations
|
||||
|
||||
* [ ] `/vex/consensus` query, detail, simulate, export.
|
||||
* [ ] Policy Engine: consume consensus signals; add thresholds and precedence.
|
||||
* [ ] Vuln Explorer: show consensus chip and triage banners; deep link to Lens.
|
||||
|
||||
### Issuer Directory
|
||||
|
||||
* [ ] CRUD for issuers and keys, audit logs, RBAC.
|
||||
* [ ] Import common CSAF publishers; seed with sane defaults.
|
||||
|
||||
### Console
|
||||
|
||||
* [ ] Build list grid with filters and saved views.
|
||||
* [ ] Quorum bar and Evidence table with signature icons and scope quality tags.
|
||||
* [ ] Conflict view for `DISPUTED`.
|
||||
* [ ] Simulation drawer integrated with Policy Studio.
|
||||
|
||||
### CLI
|
||||
|
||||
* [ ] `stella vex consensus list|show|simulate|export` with JSON/CSV.
|
||||
* [ ] Stable schemas; tests for piping and scripting.
|
||||
|
||||
### Observability/Perf
|
||||
|
||||
* [ ] Metrics, logs, traces as specified; dashboards.
|
||||
* [ ] Load tests at 10M consensus records/tenant; optimize indexes and caches.
|
||||
|
||||
### Docs
|
||||
|
||||
* [ ] Author and cross‑link all docs listed in §5.
|
||||
* [ ] Add examples and screenshots to Console doc.
|
||||
|
||||
> **Imposed rule:** Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.
|
||||
|
||||
---
|
||||
|
||||
## 7) Acceptance criteria
|
||||
|
||||
* Normalization supports CSAF VEX, OpenVEX, CycloneDX VEX with product tree mapping to purls across npm, Maven, PyPI, Go, RPM/DEB.
|
||||
* Signature verification works and affects weights; unverifiable signatures do not crash flows.
|
||||
* Consensus outputs are reproducible, explainable, and queryable at scale.
|
||||
* Vuln Explorer displays consensus state and affects policy outcomes per thresholds.
|
||||
* Simulation reflects policy trust changes without side effects and returns rationale deltas.
|
||||
* CLI/API feature parity; evidence and quorum are exportable.
|
||||
* P95 performance budgets met; dashboards reflect health.
|
||||
|
||||
---
|
||||
|
||||
## 8) Risks and mitigations
|
||||
|
||||
* **Mapping errors (CPE→purl):** use conservative scope scores, cap weights, surface warnings, manual override hooks in Policy Studio.
|
||||
* **Malicious or mistaken issuer:** signature verification plus trust weighting and tenancy overrides.
|
||||
* **Evidence storms:** debounce recompute; batch and shard workers; backpressure and prioritization.
|
||||
* **User confusion with conflicting VEX:** clear conflict UI, rationale chains, suggested actions, and policy banners.
|
||||
* **Stale statements:** recency decay and expiry windows in policy.
|
||||
|
||||
---
|
||||
|
||||
## 9) Test plan
|
||||
|
||||
* **Unit:** status mapping, comparators per ecosystem, trust weighting, threshold math.
|
||||
* **Property tests:** invariants such as monotonicity with added supporting evidence and idempotent recompute.
|
||||
* **Integration:** Excitator→Lens→Policy→Vuln Explorer pipeline with signed/unsigned, conflicting, and stale evidence.
|
||||
* **E2E Console:** list filters, detail with quorum, conflict workflows, export, simulation.
|
||||
* **Security:** RBAC on Issuer Directory, CSRF for Console, signature verification path traversal guards.
|
||||
* **Performance:** cold/hot query latencies, recompute throughput, cache hit ratios.
|
||||
* **Determinism:** time‑travel snapshots reproduce prior consensus states.
|
||||
|
||||
---
|
||||
|
||||
## 10) Philosophy
|
||||
|
||||
* **Consensus, not replacement.** The Lens summarizes without erasing dissent.
|
||||
* **Trust is contextual.** Tenants must tune weights and thresholds to their environments.
|
||||
* **Proof over prose.** Every decision comes with math, provenance, and a rationale chain.
|
||||
* **Safety by design.** When in doubt, surface conflicts instead of silently suppressing.
|
||||
|
||||
> Final reminder: **Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.**
|
||||
440
docs/implplan/EPIC_8.md
Normal file
440
docs/implplan/EPIC_8.md
Normal file
@@ -0,0 +1,440 @@
|
||||
Below is the “maximum documentation” bundle for Epic 8. It’s engineered to be pasted into your repo without turning into yet another unread wiki tomb. Slight sarcasm included to keep blood flowing.
|
||||
|
||||
> **Imposed rule:** Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.
|
||||
|
||||
---
|
||||
|
||||
# Epic 8: Advisory AI Assistant (summaries, conflict explain, remediation hints)
|
||||
|
||||
**Short name:** `Advisory AI`
|
||||
**Services touched:** Conseiller (Feedser), Excitator (Vexer), VEX Lens, SBOM Service, Policy Engine, Findings Ledger, Web API Gateway, Authority (authN/Z), Console (Web UI), CLI, Telemetry/Analytics
|
||||
**AOC ground rule:** Conseiller and Excitator aggregate but never merge or mutate source docs. Advisory AI produces derived summaries and plans with strict provenance and citations. No silent rewriting of evidence. Ever.
|
||||
|
||||
---
|
||||
|
||||
## 1) What it is
|
||||
|
||||
Advisory AI is a tenant‑scoped, retrieval‑augmented assistant that turns noisy security advisories and VEX statements into three consumable artifacts:
|
||||
|
||||
1. **Advisory Summary**
|
||||
Condenses one or more advisories (CSAF, OSV, GHSA, vendor PDFs, distro notices) into a concise brief with key facts: affected ranges, exploit status, impact, known workarounds, fixed versions, and links. Always cites the exact sources and sections used.
|
||||
|
||||
2. **Conflict Explain**
|
||||
Explains why VEX statements or advisories disagree for a specific artifact and version. Uses the VEX Consensus Lens outputs and issuer trust model to produce a human‑readable, step‑by‑step explanation: who said what, where the product scoping diverges, and what policy thresholds caused the final state.
|
||||
|
||||
3. **Remediation Hints**
|
||||
Suggests practical next steps: upgrade paths compatible with your dependency graph, backports, config toggles, temporary policy suppressions, or compensating controls. Every hint is grounded in SBOM, environment, and policy. It ships as structured JSON plus a human summary, ready to paste into a ticket.
|
||||
|
||||
It lives in the Console as a side panel, in the CLI for batch runs, and via APIs for automation. It does not change scanner results or consensus on its own. Humans remain in charge. The machine does the skimming and the math so humans can keep the judgment and the coffee.
|
||||
|
||||
---
|
||||
|
||||
## 2) Why (brief)
|
||||
|
||||
Advisories are long, inconsistent, and sometimes contradictory. Teams waste cycles reconciling PDFs with package manifests. The assistant eliminates that sludge: fast summaries, explicit conflict explanations, and remediation hints that are actually applicable to your software, not to an imaginary ideal project from 2013.
|
||||
|
||||
---
|
||||
|
||||
## 3) How it should work (maximum detail)
|
||||
|
||||
### 3.1 Capabilities
|
||||
|
||||
* **Summaries**
|
||||
|
||||
* Input: one advisory or a bundle linked by the same advisory key (CVE, GHSA, vendor‑ID), product scope, and environment.
|
||||
* Output:
|
||||
|
||||
* 150 to 300 words summary
|
||||
* `AdvisorySummary JSON` (schema below)
|
||||
* Citations with paragraph anchors
|
||||
* Confidence label and coverage score (how much of the advisory set is represented)
|
||||
* **Conflict Explain**
|
||||
|
||||
* Input: `(artifact_id, purl, version, advisory_key)` tuple.
|
||||
* Output: narrative plus a structured breakdown of consensus math, issuer votes, product mapping mismatches, and the exact policy knobs that tipped the result.
|
||||
* **Remediation Hints**
|
||||
|
||||
* Input: same tuple plus SBOM context and environment.
|
||||
* Output: ranked list of remediation options with feasibility score, blast radius estimate (derived from dependency paths), effort class, and links to fixed versions. Includes “do nothing” when the VEX consensus is not affected.
|
||||
|
||||
### 3.2 System design
|
||||
|
||||
**Architecture diagram in words (because ASCII art is a crime):**
|
||||
|
||||
1. **Retrievers**
|
||||
|
||||
* Structured retriever over Conseiller’s normalized advisory fields.
|
||||
* Vector retriever over advisory text chunks with paragraph anchors.
|
||||
* VEX retriever over Excitator evidence and VEX Lens consensus.
|
||||
* SBOM retriever for purl, version, dependency paths, env flags.
|
||||
2. **Deterministic resolvers**
|
||||
|
||||
* Version comparators per ecosystem.
|
||||
* Range satisfaction checks.
|
||||
* Dependency path scorers and blast radius estimator.
|
||||
3. **Orchestrator**
|
||||
|
||||
* Task‑specific prompt templates for Summary, Conflict, Remediation.
|
||||
* Tool calls to deterministics (version check, graph crawl) with results injected into the prompt as structured context.
|
||||
* Strict token budgets and truncation rules to avoid model babble.
|
||||
4. **Models**
|
||||
|
||||
* Default: on‑prem inference container with mid‑sized model.
|
||||
* Optional: tenant‑enabled remote inference. Disabled by default.
|
||||
* Temperature locked low for summary and conflict. Slightly higher for remediation narrative phrasing. No creativity in facts.
|
||||
5. **Guardrails**
|
||||
|
||||
* Prompt injection defense by stripping or quarantining advisory text that tries to instruct the model.
|
||||
* Fact boundary tagger. The assistant must only state facts that appear in structured inputs or cited chunks.
|
||||
* Redaction of secrets before prompts.
|
||||
* Output validator checks: required JSON fields, numeric ranges, valid version strings.
|
||||
|
||||
### 3.3 Data contracts
|
||||
|
||||
**AdvisorySummary JSON**
|
||||
|
||||
```json
|
||||
{
|
||||
"advisory_key": "CVE-2025-12345",
|
||||
"sources": [
|
||||
{"id":"csaf:vendorA:2025-001","uri":"...","sections":["2.1","3.4"]},
|
||||
{"id":"osv:pkg:npm/lodash","uri":"...","sections":["affected","references"]}
|
||||
],
|
||||
"affected_ranges": [
|
||||
{"ecosystem":"npm","purl_family":"pkg:npm/lodash","introduced":"<4.17.15","fixed": "4.17.21"}
|
||||
],
|
||||
"exploit_status": "no_known_exploit | poc_public | exploited_in_the_wild | n/a",
|
||||
"impact": {"cvss":[{"vector":"CVSS:3.1/AV:N/...","score":7.5}], "cwes":["CWE-79"]},
|
||||
"workarounds": ["Disable feature X", "Set flag Y=false"],
|
||||
"fixed_versions": ["4.17.21"],
|
||||
"notes": "Vendor states not affected on platform Z due to build option W.",
|
||||
"coverage_score": 0.86,
|
||||
"generated_at": "2025-10-25T12:00:00Z"
|
||||
}
|
||||
```
|
||||
|
||||
**ConflictExplanation JSON**
|
||||
|
||||
```json
|
||||
{
|
||||
"tuple": {"artifact_id":"svc:checkout@1.9.0","purl":"pkg:npm/lodash@4.17.20","advisory_key":"CVE-2025-12345"},
|
||||
"consensus": {"state":"NOT_AFFECTED","confidence":0.82},
|
||||
"quorum": [
|
||||
{"issuer":"lodash-maintainers","status":"NOT_AFFECTED","weight":0.9,"sig":true,"justification":"component_not_present"},
|
||||
{"issuer":"vendorX-distro","status":"AFFECTED","weight":0.25,"sig":false,"justification":"generic"}
|
||||
],
|
||||
"policy_factors": {"na_threshold":1.0,"aff_threshold":0.6,"recency_decay_days":90},
|
||||
"mapping_issues": [{"kind":"cpe_to_purl","score":0.6,"detail":"CPE wildcard matched multiple purls"}],
|
||||
"explanation_steps": [
|
||||
"Exact purl match found for maintainer VEX; weight 0.9",
|
||||
"Distro advisory generic; scope score 0.5; effective weight 0.25",
|
||||
"NA threshold met. Result set to NOT_AFFECTED"
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
**RemediationPlan JSON**
|
||||
|
||||
```json
|
||||
{
|
||||
"tuple": {"artifact_id":"svc:checkout@1.9.0","purl":"pkg:npm/lodash@4.17.20","advisory_key":"CVE-2025-12345"},
|
||||
"options": [
|
||||
{
|
||||
"kind": "upgrade",
|
||||
"target_version": "4.17.21",
|
||||
"feasibility": 0.92,
|
||||
"blast_radius": {"direct_callers":3,"transitive_depth":2},
|
||||
"effort": "low | medium | high",
|
||||
"rationale": "Semver patch, no breaking APIs in release notes",
|
||||
"links": ["release_notes_uri"]
|
||||
},
|
||||
{
|
||||
"kind": "workaround",
|
||||
"action": "Set SAFE_MODE=true",
|
||||
"feasibility": 0.6,
|
||||
"blast_radius": {"feature_flags":["SAFE_MODE"]},
|
||||
"effort": "low",
|
||||
"rationale": "Vendor states mitigation reduces attack surface on feature X"
|
||||
}
|
||||
],
|
||||
"preferred": 0,
|
||||
"policy_effects": {"sla_days": 7, "severity_override": "medium_if_not_fixed"},
|
||||
"generated_at": "2025-10-25T12:00:00Z"
|
||||
}
|
||||
```
|
||||
|
||||
### 3.4 APIs
|
||||
|
||||
```
|
||||
POST /advisory/ai/summary
|
||||
{
|
||||
"advisory_key":"CVE-2025-12345",
|
||||
"artifact_id":"svc:checkout@1.9.0",
|
||||
"purl":"pkg:npm/lodash@4.17.20",
|
||||
"sources":["csaf:*","osv:*"], // optional filters
|
||||
"policy_version":"1.3.0",
|
||||
"lang":"en"
|
||||
}
|
||||
-> 200 { "summary_text":"...", "summary": {AdvisorySummary}, "citations":[...] }
|
||||
|
||||
POST /advisory/ai/conflict
|
||||
{
|
||||
"artifact_id":"svc:checkout@1.9.0",
|
||||
"purl":"pkg:npm/lodash@4.17.20",
|
||||
"advisory_key":"CVE-2025-12345",
|
||||
"policy_version":"1.3.0"
|
||||
}
|
||||
-> 200 { "explanation_text":"...", "explanation": {ConflictExplanation} }
|
||||
|
||||
POST /advisory/ai/remediation
|
||||
{
|
||||
"artifact_id":"svc:checkout@1.9.0",
|
||||
"purl":"pkg:npm/lodash@4.17.20",
|
||||
"advisory_key":"CVE-2025-12345",
|
||||
"policy_version":"1.3.0",
|
||||
"max_options":5,
|
||||
"strategy_preference":["upgrade","backport","workaround"]
|
||||
}
|
||||
-> 200 { "plan_text":"...", "plan": {RemediationPlan} }
|
||||
|
||||
POST /advisory/ai/batch
|
||||
{
|
||||
"items":[ {tuple}, {tuple}, ... ],
|
||||
"task":"summary | conflict | remediation",
|
||||
"policy_version":"1.3.0"
|
||||
}
|
||||
-> 207 multi-status
|
||||
```
|
||||
|
||||
Status codes: `400` invalid, `403` RBAC, `404` missing evidence, `409` conflict lock, `422` output validation failed, `429` rate limit.
|
||||
|
||||
### 3.5 Console (Web UI)
|
||||
|
||||
* Surfaces:
|
||||
|
||||
* Vuln Explorer detail: “Advisory AI” side panel with 3 tabs: Summary, Conflict, Remediation.
|
||||
* Consensus Lens detail: prominent “Explain conflict” button.
|
||||
* Policy Studio sim: “Show effect on assistant output” preview.
|
||||
|
||||
* UX details:
|
||||
|
||||
* Citations are footnotes with hover to show source paragraph.
|
||||
* “Copy as ticket” produces Markdown and JSON.
|
||||
* Plan options show feasibility bar, blast radius chips, and required approvals per policy.
|
||||
* Injection warnings appear if advisory text included unsafe instructions.
|
||||
|
||||
* A11y: ARIA tags for tabs, keyboard shortcuts `G` to generate, `R` to refresh, `C` to copy JSON.
|
||||
|
||||
### 3.6 CLI
|
||||
|
||||
```
|
||||
stella advise summarize --advisory CVE-2025-12345 --artifact svc:checkout@1.9.0 --purl pkg:npm/lodash@4.17.20 --policy 1.3.0 --json
|
||||
stella advise explain --advisory CVE-2025-12345 --artifact svc:checkout@1.9.0 --purl pkg:npm/lodash@4.17.20 --policy 1.3.0
|
||||
stella advise remediate --advisory CVE-2025-12345 --artifact svc:checkout@1.9.0 --purl pkg:npm/lodash@4.17.20 --policy 1.3.0 --strategy upgrade,workaround --out plan.json
|
||||
stella advise batch --file tuples.json --task remediation --policy 1.3.0
|
||||
```
|
||||
|
||||
Exit codes: `0` ok, `2` invalid args, `4` not found, `5` denied, `7` validation fail.
|
||||
|
||||
### 3.7 RBAC and security
|
||||
|
||||
* Roles:
|
||||
|
||||
* Viewer can run summaries and read explanations.
|
||||
* Operator can run remediation and export plans.
|
||||
* Admin can toggle model endpoints and guardrail settings.
|
||||
|
||||
* Defaults:
|
||||
|
||||
* Remote model calls disabled.
|
||||
* Redaction on.
|
||||
* Prompt logging anonymized.
|
||||
* Outputs stored as derived artifacts with TTL (default 30 days) unless pinned to a ticket.
|
||||
|
||||
### 3.8 Observability
|
||||
|
||||
* Metrics:
|
||||
|
||||
* `advisory_ai_latency_ms` by task type.
|
||||
* `advisory_ai_guardrail_blocks_total`.
|
||||
* `advisory_ai_output_validation_fail_total`.
|
||||
* `advisory_ai_citation_coverage` gauge.
|
||||
* Traces: retriever spans, tool calls, model inference, validator.
|
||||
* Logs: include tuple key, token usage, truncation events, and guardrail outcomes.
|
||||
|
||||
### 3.9 Performance
|
||||
|
||||
* Targets:
|
||||
|
||||
* P95 under 1.5 s for Summary and Conflict with warm caches.
|
||||
* P95 under 2.5 s for Remediation on medium SBOMs (1000 packages).
|
||||
* Batch throughput 10 tuples per second per worker.
|
||||
|
||||
### 3.10 Edge cases
|
||||
|
||||
* Advisory missing fixed versions: produce workaround‑only plan and mark feasibility low.
|
||||
* Conflicts with near‑tie weights: declare “DISPUTED” and require human approval, no auto plan preferred.
|
||||
* Exotic version schemes: fallback to string compare with warning and feasibility cap.
|
||||
* Private packages: no public release notes. Prefer internal changelog links if attached to artifact metadata.
|
||||
* Multi‑env differences: render per‑env deltas when policy knobs differ (dev vs prod).
|
||||
|
||||
---
|
||||
|
||||
## 4) Implementation plan
|
||||
|
||||
### 4.1 Services and components
|
||||
|
||||
* **New:** `src/AdvisoryAI/StellaOps.AdvisoryAI`
|
||||
|
||||
* `retriever/` wrappers for Conseiller, Excitator, VEX Lens, SBOM.
|
||||
* `deterministic/` version and path analyzers.
|
||||
* `orchestrator/` task routers and prompt builders.
|
||||
* `guardrails/` injection, redaction, output validator.
|
||||
* `api/` REST endpoints and schema enforcement.
|
||||
|
||||
* **Updates:**
|
||||
|
||||
* Conseiller: expose paragraph‑level anchors for advisories.
|
||||
* Excitator: expose justifications and product trees in normalized form.
|
||||
* VEX Lens: stable API for quorum and rationale.
|
||||
* SBOM Service: efficient path queries and versions timeline per purl.
|
||||
|
||||
### 4.2 Packaging
|
||||
|
||||
* Container images:
|
||||
|
||||
* `stella/advisory-ai:<<version>>`
|
||||
* `stella/inference:<<version>>` (if using on‑prem model)
|
||||
* Helm values to toggle remote inference and GPU.
|
||||
|
||||
### 4.3 Rollout
|
||||
|
||||
* Phase 1: Summary and Conflict read‑only.
|
||||
* Phase 2: Remediation with “Copy as ticket”.
|
||||
* Phase 3: Batch APIs, CLI, and Policy Studio simulation hooks.
|
||||
|
||||
---
|
||||
|
||||
## 5) Documentation changes
|
||||
|
||||
Create or update the following files. Each doc ends with the imposed rule statement.
|
||||
|
||||
1. `/docs/advisory-ai/overview.md`
|
||||
What it is, capabilities, guardrails, AOC alignment, RBAC.
|
||||
|
||||
2. `/docs/advisory-ai/architecture.md`
|
||||
RAG design, retrievers, orchestrator, deterministics, models, caching.
|
||||
|
||||
3. `/docs/advisory-ai/api.md`
|
||||
Endpoint specs, payload schemas, error codes, examples.
|
||||
|
||||
4. `/docs/advisory-ai/console.md`
|
||||
Screens, actions, a11y, how citations work, copy‑as‑ticket.
|
||||
|
||||
5. `/docs/advisory-ai/cli.md`
|
||||
Command usage, exit codes, piping examples.
|
||||
|
||||
6. `/docs/policy/assistant-parameters.md`
|
||||
Temperature, max tokens, plan ranking weights, TTLs.
|
||||
|
||||
7. `/docs/security/assistant-guardrails.md`
|
||||
Redaction rules, injection defense, output validation, logging.
|
||||
|
||||
8. `/docs/sbom/remediation-heuristics.md`
|
||||
Feasibility scoring, blast radius, effort classes.
|
||||
|
||||
9. `/docs/runbooks/assistant-ops.md`
|
||||
Warmup, cache priming, model outages, scaling, on‑call steps.
|
||||
|
||||
---
|
||||
|
||||
## 6) Engineering tasks
|
||||
|
||||
### Backend core
|
||||
|
||||
* [ ] Implement structured and vector retrievers with paragraph anchors from Conseiller.
|
||||
* [ ] Implement VEX retriever using Lens APIs with caching.
|
||||
* [ ] Build deterministics: ecosystem comparators, range checks, dependency path scorer.
|
||||
* [ ] Implement orchestrator with task‑specific templates and tool call pipeline.
|
||||
* [ ] Implement guardrails and validators with hard failure on invalid JSON.
|
||||
* [ ] Add RBAC to endpoints and anonymized prompt logging.
|
||||
* [ ] Add caching layer with tuple‑keyed entries and policy version scoping.
|
||||
|
||||
### Integrations
|
||||
|
||||
* [ ] Conseiller: expose advisory chunk API and metadata needed for citations.
|
||||
* [ ] Excitator: ensure justifications and product trees are queryable.
|
||||
* [ ] VEX Lens: add “policy factors” endpoint for explanation rendering.
|
||||
* [ ] SBOM Service: implement `GET /sbom/paths?purl=...` and version timeline.
|
||||
|
||||
### Console
|
||||
|
||||
* [ ] Build Advisory AI panel with 3 tabs and citation tooltips.
|
||||
* [ ] Implement “Copy as ticket” (Markdown + JSON) and download.
|
||||
* [ ] Add injection warning banner when triggered.
|
||||
* [ ] Respect a11y requirements and shortcuts.
|
||||
|
||||
### CLI
|
||||
|
||||
* [ ] `stella advise summarize|explain|remediate|batch` with JSON output.
|
||||
* [ ] Add `--out` option to save plans and summaries.
|
||||
* [ ] Tests for piping and jq workflows.
|
||||
|
||||
### Observability
|
||||
|
||||
* [ ] Emit metrics and traces listed in §3.8.
|
||||
* [ ] Dashboards: latency, guardrail blocks, validation fails, coverage.
|
||||
|
||||
### Docs
|
||||
|
||||
* [ ] Write all files in §5 with examples and screenshots.
|
||||
* [ ] Cross‑link to VEX Lens and Vulnerability Explorer docs.
|
||||
|
||||
> **Imposed rule:** Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.
|
||||
|
||||
---
|
||||
|
||||
## 7) Acceptance criteria
|
||||
|
||||
* Summaries cite specific source sections and reflect affected ranges and fixed versions correctly for at least 95% of a validation set.
|
||||
* Conflict explanations enumerate issuers, weights, justifications, mapping issues, and policy thresholds that caused the consensus state.
|
||||
* Remediation plans output at least one feasible option when a fixed version exists and correctly flag “no public fix” cases.
|
||||
* JSON schemas validate for all outputs.
|
||||
* Console shows the panel with citations, copy‑as‑ticket, and a11y passes.
|
||||
* CLI produces identical JSON to API responses.
|
||||
* Guardrails block injection attempts and redact secrets in prompts.
|
||||
* P95 latency targets are met with warm caches.
|
||||
* No mutation of raw advisory or VEX evidence occurs anywhere in the pipeline.
|
||||
|
||||
---
|
||||
|
||||
## 8) Risks and mitigations
|
||||
|
||||
* **Prompt injection in advisory text.** Strip instructions, sandbox chunks, and highlight to user when removed.
|
||||
* **Hallucinated facts.** Hard validation requires facts to appear in structured inputs or cited text. Fail closed if not provable.
|
||||
* **Mapping errors produce bad hints.** Depend on SBOM Graph and VEX Lens scope scores; cap feasibility when scope is weak.
|
||||
* **Model outage.** Degrade to deterministic summaries (shorter, but accurate).
|
||||
* **Privacy concerns.** Default on‑prem inference, remote endpoints opt‑in with clear flags and audit logs.
|
||||
|
||||
---
|
||||
|
||||
## 9) Test plan
|
||||
|
||||
* **Unit:** version comparators, range checks, feasibility scoring, output validators.
|
||||
* **Golden files:** advisory sets mapped to expected summaries and plans; diff on each build.
|
||||
* **Injection tests:** adversarial advisories with “ignore prior instructions” payloads must be neutralized.
|
||||
* **Integration:** Conseiller→Advisory AI→Console loop, with VEX Lens conflicts and SBOM graph lookups.
|
||||
* **E2E:** generate summary, explanation, and plan for representative ecosystems (npm, Maven, PyPI, Go, RPM/DEB).
|
||||
* **Perf:** soak tests with 5k tuples batch; observe cache hit ratios and P95.
|
||||
* **A11y:** keyboard navigation and screen reader labels.
|
||||
|
||||
---
|
||||
|
||||
## 10) Philosophy
|
||||
|
||||
* **Facts first.** If it is not in structured inputs or citations, it does not exist.
|
||||
* **Explain everything.** Humans should see exactly why the tool said what it said.
|
||||
* **Helpful by default.** Plans must consider the real dependency graph and environment, not fantasy.
|
||||
* **No silent merges.** Evidence is sacred. Summaries and plans are separate, auditable derivatives.
|
||||
|
||||
> Final reminder: **Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.**
|
||||
523
docs/implplan/EPIC_9.md
Normal file
523
docs/implplan/EPIC_9.md
Normal file
@@ -0,0 +1,523 @@
|
||||
Below is the “maximum documentation” bundle for Epic 9. Paste it into your repo and pretend the ingestion chaos was always under control.
|
||||
|
||||
> **Imposed rule:** Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.
|
||||
|
||||
---
|
||||
|
||||
# Epic 9: Source & Job Orchestrator Dashboard
|
||||
|
||||
**Short name:** `Orchestrator Dashboard`
|
||||
**Primary service:** `orchestrator` (scheduler, queues, rate‑limits, job state)
|
||||
**Surfaces:** Console (Web UI), CLI, Web API
|
||||
**Touches:** Conseiller (Feedser), Excitator (Vexer), VEX Consensus Lens, SBOM Service, Policy Engine, Findings Ledger, Authority (authN/Z), Telemetry/Analytics, Object Storage, Relational DB, Message Bus
|
||||
|
||||
**AOC ground rule:** Conseiller and Excitator aggregate but never merge. The orchestrator schedules, tracks and recovers jobs; it does not transform evidence beyond transport and storage. No “smart” merging in flight.
|
||||
|
||||
---
|
||||
|
||||
## 1) What it is
|
||||
|
||||
The Source & Job Orchestrator Dashboard is the control surface for every data source and pipeline run across StellaOps. It gives operators:
|
||||
|
||||
* Live health of all advisory/VEX/SBOM sources and derived jobs.
|
||||
* End‑to‑end pipeline visibility as DAGs and timelines.
|
||||
* Controls for pausing, backfilling, replaying, throttling and retrying.
|
||||
* Error pattern analysis, rate‑limit observability and backpressure insights.
|
||||
* Provenance and audit trails from initial fetch through parse, normalize, index and policy evaluation.
|
||||
|
||||
The dashboard sits over the `orchestrator` service, which maintains job state, schedules runs, enforces quotas and rate limits, and collects metrics from worker pools embedded in Conseiller, Excitator, SBOM and related services.
|
||||
|
||||
---
|
||||
|
||||
## 2) Why (brief)
|
||||
|
||||
Ingestion breaks quietly and then loudly. Without a unified control plane, you learn about it from angry users or empty indexes. This dashboard shortens incident MTTR, enables safe backfills, and makes compliance reviewers stop sending emails with twelve attachments and one emoji.
|
||||
|
||||
---
|
||||
|
||||
## 3) How it should work (maximum detail)
|
||||
|
||||
### 3.1 Capabilities
|
||||
|
||||
* **Source registry**
|
||||
|
||||
* Register, tag and version connectors (OSV, GHSA, CSAF endpoints, vendor PDF scrapers, distro feeds, RSS, S3 drops, internal registries).
|
||||
* Store connection details, secrets (via KMS), rate‑limit policy, schedules, and ownership metadata.
|
||||
* Validate and “test connection” safely.
|
||||
|
||||
* **Job orchestration**
|
||||
|
||||
* Create DAGs composed of job types: `fetch`, `parse`, `normalize`, `dedupe`, `index`, `consensus_compute`, `policy_eval`, `crosslink`, `sbom_ingest`, `sbom_index`.
|
||||
* Priorities, queues, concurrency caps, exponential backoff, circuit breakers.
|
||||
* Idempotency keys and output artifact hashing to avoid duplicate work.
|
||||
* Event‑time watermarks for backfills without double counting.
|
||||
|
||||
* **Observability & control**
|
||||
|
||||
* Gantt timeline and real‑time DAG view with critical path highlighting.
|
||||
* Backpressure and queue depth heatmaps.
|
||||
* Error clustering by class (HTTP 429, TLS, schema mismatch, parse failure, upstream 5xx).
|
||||
* Per‑source SLOs and SLA budgets with burn‑rate alerts.
|
||||
* One‑click actions: retry, replay range, pause/resume, throttle/unthrottle, reroute to canary workers.
|
||||
|
||||
* **Provenance & audit**
|
||||
|
||||
* Immutable run ledger linking input artifact → every job → output artifact.
|
||||
* Schema version tracking and drift detection.
|
||||
* Operator actions recorded with reason and ticket reference.
|
||||
|
||||
* **Safety**
|
||||
|
||||
* Secret redaction everywhere.
|
||||
* Tenant isolation at API, queue and storage layers.
|
||||
* AOC: no in‑flight merges of advisory or VEX content.
|
||||
|
||||
### 3.2 Core architecture
|
||||
|
||||
* **orchestrator (service)**
|
||||
|
||||
* Maintains job state in Postgres (`sources`, `runs`, `jobs`, `artifacts`, `dag_edges`, `quotas`, `schedules`).
|
||||
* Publishes work to a message bus (e.g., `topic.jobs.ready.<queue>`).
|
||||
* Distributed token‑bucket rate limiter per source/tenant/host.
|
||||
* Watchdog for stuck jobs and circuit breakers for flapping sources.
|
||||
* Watermark manager for backfills (event‑time windows).
|
||||
|
||||
* **worker SDK**
|
||||
|
||||
* Lightweight library embedded in Conseiller/Excitator/SBOM workers to:
|
||||
|
||||
* Claim work, heartbeat, update progress, report metrics.
|
||||
* Emit artifact metadata and checksums.
|
||||
* Enforce idempotency via orchestrator‑supplied key.
|
||||
|
||||
* **object store**
|
||||
|
||||
* Raw payloads and intermediate artifacts organized by schema and hash:
|
||||
|
||||
* `advisory/raw/<source_id>/<event_time>/<sha256>.json|pdf`
|
||||
* `advisory/normalized/<schema_ver>/<hash>.json`
|
||||
* `vex/raw|normalized/...`
|
||||
* `sbom/raw|graph/...`
|
||||
|
||||
* **web API**
|
||||
|
||||
* CRUD for sources, runs, jobs, schedules, quotas.
|
||||
* Control actions (retry, cancel, pause, backfill).
|
||||
* Streaming updates via WebSocket/SSE for the Console.
|
||||
|
||||
* **console**
|
||||
|
||||
* React app consuming Orchestrator APIs, rendering DAGs, timelines, health charts and action panels with RBAC.
|
||||
|
||||
### 3.3 Data model (selected tables)
|
||||
|
||||
* `sources`
|
||||
|
||||
* `id`, `kind` (`advisory|vex|sbom|internal`), `subtype` (e.g., `osv`, `ghsa`, `csaf`, `vendor_pdf`), `display_name`, `owner_team`, `schedule_cron`, `rate_policy`, `enabled`, `secrets_ref`, `tags`, `schema_hint`, `created_at`, `updated_at`.
|
||||
|
||||
* `runs`
|
||||
|
||||
* `id`, `source_id`, `trigger` (`schedule|manual|event|backfill`), `window_start`, `window_end`, `state`, `started_at`, `finished_at`, `stats_json`.
|
||||
|
||||
* `jobs`
|
||||
|
||||
* `id`, `run_id`, `type`, `queue`, `priority`, `state` (`pending|running|succeeded|failed|canceled|deadletter`), `attempt`, `max_attempt`, `idempotency_key`, `input_artifact_id`, `output_artifact_id`, `worker_id`, `created_at`, `started_at`, `finished_at`, `error_class`, `error_message`, `metrics_json`.
|
||||
|
||||
* `dag_edges`
|
||||
|
||||
* `from_job_id`, `to_job_id`, `edge_kind` (`success_only|always`).
|
||||
|
||||
* `artifacts`
|
||||
|
||||
* `id`, `kind` (`raw|normalized|index|consensus`), `schema_ver`, `hash`, `uri`, `bytes`, `meta_json`, `created_at`.
|
||||
|
||||
* `quotas`
|
||||
|
||||
* `tenant_id`, `resource` (`requests_per_min`, `concurrent_jobs`), `limit`, `window_sec`.
|
||||
|
||||
* `schedules`
|
||||
|
||||
* Per‑source cron plus jitter, timezone, blackout windows.
|
||||
|
||||
### 3.4 Job lifecycle
|
||||
|
||||
1. **Plan**
|
||||
Scheduler creates a `run` for a source and plans a DAG: e.g., `fetch → parse → normalize → dedupe → index → policy_eval` (advisory) or `fetch → parse → normalize → consensus_compute` (VEX).
|
||||
|
||||
2. **Enqueue**
|
||||
Ready nodes become `jobs` with queue, priority, idempotency key and optional rate‑limit tokens reserved.
|
||||
|
||||
3. **Execute**
|
||||
Worker claims job, heartbeats every N seconds. Output artifacts are stored and linked. Failures are classified and retried with exponential backoff and jitter, up to `max_attempt`.
|
||||
|
||||
4. **Complete**
|
||||
Downstream nodes unblock. On run completion, orchestrator computes SLO deltas and emits run summary.
|
||||
|
||||
5. **Dead‑letter**
|
||||
Jobs exceeding attempts move to a DLQ with structured context and suggested remediation.
|
||||
|
||||
### 3.5 Scheduling, backpressure, rate‑limits
|
||||
|
||||
* **Token bucket** per `{tenant, source.host}` with adaptive refill if upstream 429/503 seen.
|
||||
* **Concurrency caps** per source and per job type to avoid thundering herd.
|
||||
* **Backpressure signals** from queue depth, worker CPU, and upstream error rates; scheduler reduces inflight issuance accordingly.
|
||||
* **Backfills** use event‑time windows with immutable watermarks to avoid re‑processing.
|
||||
* **Blackout windows** for vendor maintenance periods.
|
||||
|
||||
### 3.6 APIs
|
||||
|
||||
```
|
||||
POST /orchestrator/sources
|
||||
GET /orchestrator/sources?kind=&tag=&q=
|
||||
GET /orchestrator/sources/{id}
|
||||
PATCH /orchestrator/sources/{id}
|
||||
POST /orchestrator/sources/{id}/actions:test|pause|resume|sync-now
|
||||
POST /orchestrator/sources/{id}/backfill { "from":"2024-01-01", "to":"2024-03-01" }
|
||||
|
||||
GET /orchestrator/runs?source_id=&state=&from=&to=
|
||||
GET /orchestrator/runs/{run_id}
|
||||
GET /orchestrator/runs/{run_id}/dag
|
||||
POST /orchestrator/runs/{run_id}/cancel
|
||||
|
||||
GET /orchestrator/jobs?state=&type=&queue=&source_id=
|
||||
GET /orchestrator/jobs/{job_id}
|
||||
POST /orchestrator/jobs/{job_id}/actions:retry|cancel|prioritize
|
||||
|
||||
GET /orchestrator/metrics/overview
|
||||
GET /orchestrator/errors/top?window=1h
|
||||
GET /orchestrator/quotas
|
||||
PATCH /orchestrator/quotas/{tenant_id}
|
||||
WS /orchestrator/streams/updates
|
||||
```
|
||||
|
||||
### 3.7 Console (Web UI)
|
||||
|
||||
* **Overview**
|
||||
|
||||
* KPI tiles: sources healthy, runs in progress, queue depth, error rate, burn‑rate to SLO.
|
||||
* Heatmap of source health by last 24h success ratio.
|
||||
|
||||
* **Sources**
|
||||
|
||||
* Grid with filters, inline status (active, paused, throttled), next run eta, last error class.
|
||||
* Detail panel: config, secrets status (redacted), schedule, rate limits, ownership, run history, action buttons.
|
||||
|
||||
* **Runs**
|
||||
|
||||
* Timeline (Gantt) with critical path, duration distribution, and per‑stage breakdown.
|
||||
* Run detail: DAG view with node metrics, artifacts, logs, action menu (cancel).
|
||||
|
||||
* **Jobs**
|
||||
|
||||
* Live table with state filters and “tail” view.
|
||||
* Job detail: payload preview (redacted), worker, attempts, stack traces, linked artifacts.
|
||||
|
||||
* **Errors**
|
||||
|
||||
* Clusters by class and signature, suggested remediations (pause source, lower concurrency, patch parser).
|
||||
|
||||
* **Queues & Backpressure**
|
||||
|
||||
* Per‑queue depth, service rate, inflight, age percentiles.
|
||||
* Rate‑limit tokens graphs per source host.
|
||||
|
||||
* **Controls**
|
||||
|
||||
* Backfill wizard with event‑time preview and safety checks.
|
||||
* Canary routing: route 5% of next 100 runs to a new worker pool.
|
||||
|
||||
* **A11y**
|
||||
|
||||
* Keyboard nav, ARIA roles for DAG nodes, live regions for updates, color‑blind friendly graphs.
|
||||
|
||||
### 3.8 CLI
|
||||
|
||||
```
|
||||
stella orch sources list --kind advisory --tag prod
|
||||
stella orch sources add --file source.yaml
|
||||
stella orch sources test <source-id>
|
||||
stella orch sources pause <source-id> # or resume
|
||||
stella orch sources sync-now <source-id>
|
||||
stella orch sources backfill <source-id> --from 2024-01-01 --to 2024-03-01
|
||||
|
||||
stella orch runs list --source <id> --state running
|
||||
stella orch runs show <run-id> --dag
|
||||
stella orch runs cancel <run-id>
|
||||
|
||||
stella orch jobs list --state failed --type parse --limit 100
|
||||
stella orch jobs retry <job-id>
|
||||
stella orch jobs cancel <job-id>
|
||||
stella orch jobs tail --queue normalize --follow
|
||||
|
||||
stella orch quotas get --tenant default
|
||||
stella orch quotas set --tenant default --concurrent-jobs 50 --rpm 1200
|
||||
```
|
||||
|
||||
Exit codes: `0` success, `2` invalid args, `4` not found, `5` denied, `7` precondition failed, `8` rate‑limited.
|
||||
|
||||
### 3.9 RBAC & security
|
||||
|
||||
* **Roles**
|
||||
|
||||
* `Orch.Viewer`: read‑only sources/runs/jobs/metrics.
|
||||
* `Orch.Operator`: perform actions on sources and jobs, launch backfills.
|
||||
* `Orch.Admin`: manage quotas, schedules, connector versions, and delete sources.
|
||||
|
||||
* **Secrets**
|
||||
|
||||
* Stored only as references to your KMS; never persisted in cleartext.
|
||||
* Console shows redact badges and last rotated timestamp.
|
||||
|
||||
* **Tenancy**
|
||||
|
||||
* Source, run, job rows scoped by tenant id.
|
||||
* Queue names and token buckets namespaced per tenant.
|
||||
|
||||
* **Compliance**
|
||||
|
||||
* Full audit log for every operator action with “reason” and optional ticket link.
|
||||
* Exportable run ledger for audits.
|
||||
|
||||
### 3.10 Observability
|
||||
|
||||
* **Metrics (examples)**
|
||||
|
||||
* `orch_jobs_inflight{type,queue}`
|
||||
* `orch_jobs_latency_ms{type,percentile}`
|
||||
* `orch_rate_tokens_available{source}`
|
||||
* `orch_error_rate{source,error_class}`
|
||||
* `orch_slo_burn_rate{source,slo}`
|
||||
* `orch_deadletter_total{source,type}`
|
||||
|
||||
* **Traces**
|
||||
|
||||
* Span per job with baggage: `run_id`, `source_id`, `artifact_id`.
|
||||
* Links across services to Conseiller/Excitator/SBOM workers.
|
||||
|
||||
* **Logs**
|
||||
|
||||
* Structured JSON with correlation ids, attempt numbers and redacted payload previews.
|
||||
|
||||
### 3.11 Performance targets
|
||||
|
||||
* Job dispatch P95 < 150 ms after dependency satisfied.
|
||||
* Scheduler loop P95 < 500 ms for 10k pending jobs.
|
||||
* Console live updates sub‑second at 1k events/sec per tenant.
|
||||
* Backfill throughput ≥ 200 jobs/sec per worker pool with zero dupes.
|
||||
|
||||
### 3.12 Edge cases & behaviors
|
||||
|
||||
* **Upstream 429 storms:** auto‑throttle, pause optional, recommend extended jitter.
|
||||
* **Schema drift:** parser moves job to DLQ with `error_class=schema_mismatch` and opens a change ticket via webhook.
|
||||
* **Flapping source:** circuit breaker opens after N consecutive failures; requires human “resume”.
|
||||
* **Clock skew:** watermark logic uses upstream event time; large skews flagged.
|
||||
* **Idempotency collisions:** new attempt yields no‑op if artifact hash already exists.
|
||||
|
||||
---
|
||||
|
||||
## 4) Implementation plan
|
||||
|
||||
### 4.1 Modules (new and updated)
|
||||
|
||||
* New service: `src/Orchestrator/StellaOps.Orchestrator`
|
||||
|
||||
* `api/` REST + WS handlers
|
||||
* `scheduler/` run planner, DAG builder, watermark/backfill logic
|
||||
* `queues/` publisher and consumer abstractions
|
||||
* `ratelimit/` token bucket and adaptive controller
|
||||
* `state/` Postgres repositories and migrations
|
||||
* `audit/` action logging and export
|
||||
* `metrics/` Prometheus exporters
|
||||
* `security/` tenant scoping, KMS client, secret refs
|
||||
|
||||
* Worker SDKs:
|
||||
|
||||
* `src/Orchestrator/StellaOps.Orchestrator.WorkerSdk.Go` and `src/Orchestrator/StellaOps.Orchestrator.WorkerSdk.Python` with job claim, heartbeat, progress, artifact publish, and structured error reporting.
|
||||
|
||||
* Console:
|
||||
|
||||
* `console/apps/orch/` pages: Overview, Sources, Runs, Jobs, Errors, Queues.
|
||||
* `components/dag-view/`, `components/gantt/`, `components/health-heatmap/`.
|
||||
|
||||
* Updates to existing services:
|
||||
|
||||
* Conseiller/Excitator/SBOM workers adopt SDK and emit artifacts with schema/version/fingerprint.
|
||||
* VEX Lens exposes `consensus_compute` as a jobable operation.
|
||||
* Policy Engine exposes `policy_eval` as a job type for scheduled recalcs.
|
||||
|
||||
### 4.2 Packaging & deployment
|
||||
|
||||
* Containers:
|
||||
|
||||
* `stella/orchestrator:<ver>`
|
||||
* `stella/worker-sdk-examples:<ver>` for canary pools
|
||||
|
||||
* Helm values:
|
||||
|
||||
* Queues/topics, per‑tenant concurrency, rate‑limit defaults, WS replica count.
|
||||
* KMS integration secrets.
|
||||
|
||||
* Migrations:
|
||||
|
||||
* Flyway/Goose migrations for new tables and indexes.
|
||||
|
||||
### 4.3 Rollout strategy
|
||||
|
||||
* Phase 1: Read‑only dashboard fed by existing job tables; no controls.
|
||||
* Phase 2: Control actions enabled for non‑prod tenants.
|
||||
* Phase 3: Backfills and quota management, then GA.
|
||||
|
||||
---
|
||||
|
||||
## 5) Documentation changes
|
||||
|
||||
Create/update the following, each ending with the imposed rule statement.
|
||||
|
||||
1. `/docs/orchestrator/overview.md`
|
||||
Concepts, roles, responsibilities, AOC alignment.
|
||||
|
||||
2. `/docs/orchestrator/architecture.md`
|
||||
Scheduler, DAGs, watermarks, queues, rate‑limits, data model.
|
||||
|
||||
3. `/docs/orchestrator/api.md`
|
||||
Endpoints, WebSocket events, error codes, examples.
|
||||
|
||||
4. `/docs/orchestrator/console.md`
|
||||
Screens, actions, a11y, live updates.
|
||||
|
||||
5. `/docs/orchestrator/cli.md`
|
||||
Commands, examples, exit codes, scripting patterns.
|
||||
|
||||
6. `/docs/orchestrator/run‑ledger.md`
|
||||
Provenance and audit export format.
|
||||
|
||||
7. `/docs/security/secrets‑handling.md`
|
||||
KMS references, redaction rules, operator hygiene.
|
||||
|
||||
8. `/docs/operations/orchestrator‑runbook.md`
|
||||
Common failures, backfill guide, circuit breakers, tuning.
|
||||
|
||||
9. `/docs/schemas/artifacts.md`
|
||||
Artifact kinds, schema versions, hashing, storage layout.
|
||||
|
||||
10. `/docs/slo/orchestrator‑slo.md`
|
||||
SLO definitions, measurement, alerting.
|
||||
|
||||
---
|
||||
|
||||
## 6) Engineering tasks
|
||||
|
||||
### Backend (orchestrator)
|
||||
|
||||
* [ ] Stand up Postgres schemas and indices for sources, runs, jobs, dag_edges, artifacts, quotas, schedules.
|
||||
* [ ] Implement scheduler: DAG planner, dependency resolver, critical path computation.
|
||||
* [ ] Implement rate limiter with adaptive behavior on 429/503 and per‑tenant tokens.
|
||||
* [ ] Implement watermark/backfill manager with event‑time windows and idempotency keys.
|
||||
* [ ] Implement API endpoints + OpenAPI spec + request validation.
|
||||
* [ ] Implement WebSocket/SSE event stream for live updates.
|
||||
* [ ] Implement audit logging and export.
|
||||
* [ ] Implement dead‑letter store and replay.
|
||||
|
||||
### Worker SDKs and integrations
|
||||
|
||||
* [ ] Build Go/Python SDKs with claim/heartbeat/progress API.
|
||||
* [ ] Integrate SDK into Conseiller, Excitator, SBOM workers; ensure artifact emission with schema ver.
|
||||
* [ ] Add `consensus_compute` and `policy_eval` as job types with deterministic inputs/outputs.
|
||||
|
||||
### Console
|
||||
|
||||
* [ ] Overview tiles and health heatmap.
|
||||
* [ ] Source list/detail with actions and config view.
|
||||
* [ ] Runs timeline (Gantt) and DAG visualization with node inspector.
|
||||
* [ ] Jobs “tail” with live updates and filters.
|
||||
* [ ] Errors clustering and suggested remediations.
|
||||
* [ ] Queues/backpressure dashboard.
|
||||
* [ ] Backfill wizard with safety checks.
|
||||
|
||||
### Observability
|
||||
|
||||
* [ ] Emit metrics listed in §3.10 and wire traces across services.
|
||||
* [ ] Dashboards: health, queue depth, error classes, burn‑rate, dispatch latency.
|
||||
* [ ] Alerts for SLO burn and circuit breaker opens.
|
||||
|
||||
### Security & RBAC
|
||||
|
||||
* [ ] Enforce tenant scoping on all endpoints; test leakage.
|
||||
* [ ] Wire KMS for secret refs and redact everywhere.
|
||||
* [ ] Implement `Orch.Viewer|Operator|Admin` roles and check in Console and API.
|
||||
|
||||
### Docs
|
||||
|
||||
* [ ] Author all files in §5 with examples and screenshots.
|
||||
* [ ] Cross‑link from Conseiller/Excitator/SBOM pages to the dashboard docs.
|
||||
* [ ] Append imposed rule to each page.
|
||||
|
||||
> **Imposed rule:** Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.
|
||||
|
||||
---
|
||||
|
||||
## 7) Acceptance criteria
|
||||
|
||||
* Operators can: pause/resume a source, run “sync‑now,” initiate a backfill for a date range, and retry/cancel individual jobs from Console and CLI.
|
||||
* DAG and timeline reflect reality within 1 second of job state changes at P95.
|
||||
* Backfills do not create duplicate artifacts; idempotency proven by hash equality.
|
||||
* Rate limiter reduces 429s by ≥80% under simulated throttle tests.
|
||||
* Audit log includes who/when/why for every operator action.
|
||||
* Provenance ledger exports a complete chain for any artifact.
|
||||
* RBAC prevents non‑admins from quota changes; tenancy isolation proven via automated tests.
|
||||
* SLO dashboard shows burn‑rate and triggers alerts under injected failure.
|
||||
|
||||
---
|
||||
|
||||
## 8) Risks & mitigations
|
||||
|
||||
* **Orchestrator becomes a single bottleneck.**
|
||||
Horizontal scale stateless workers; DB indexes tuned; job state updates batched; cache hot paths.
|
||||
|
||||
* **Secret spillage.**
|
||||
Only KMS references stored; aggressive redaction; log scrubbing in SDK.
|
||||
|
||||
* **Over‑eager backfills overwhelm upstream.**
|
||||
Enforce per‑source quotas and sandbox previews; dry‑run backfills first.
|
||||
|
||||
* **Schema drift silently corrupts normalization.**
|
||||
Hard‑fail on mismatch; DLQ with clear signatures; schema registry gating.
|
||||
|
||||
* **Flapping sources cause alert fatigue.**
|
||||
Circuit breaker with cool‑down and deduped alerts; error budget policy.
|
||||
|
||||
---
|
||||
|
||||
## 9) Test plan
|
||||
|
||||
* **Unit**
|
||||
Scheduler DAG building, topological sort, backoff math, token bucket, watermark math.
|
||||
|
||||
* **Integration**
|
||||
Orchestrator ↔ worker SDK, artifact store wiring, DLQ replay, audit pipeline.
|
||||
|
||||
* **Chaos**
|
||||
Inject 429 storms, packet loss, worker crashes; verify throttling and recovery.
|
||||
|
||||
* **Backfill**
|
||||
Simulate overlapping windows and verify idempotency and watermark correctness.
|
||||
|
||||
* **Perf**
|
||||
10k concurrent jobs: dispatch latency, DB contention, WebSocket fan‑out.
|
||||
|
||||
* **Security**
|
||||
Multi‑tenant isolation tests; KMS mock tests for secret access; RBAC matrix.
|
||||
|
||||
* **UX/A11y**
|
||||
Screen reader labels on DAG, keyboard navigation, live region updates.
|
||||
|
||||
---
|
||||
|
||||
## 10) Philosophy
|
||||
|
||||
* **Make the invisible visible.** Pipelines should be legible at a glance.
|
||||
* **Prefer reproducibility to heroics.** Idempotency and provenance over “we think it ran.”
|
||||
* **Safeguards before speed.** Throttle first, retry thoughtfully, never melt upstreams.
|
||||
* **No silent merges.** Evidence remains immutable; transformations are explicit, logged and reversible.
|
||||
|
||||
> Final reminder: **Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.**
|
||||
1831
docs/implplan/EXECPLAN.md
Normal file
1831
docs/implplan/EXECPLAN.md
Normal file
File diff suppressed because it is too large
Load Diff
1096
docs/implplan/SPRINTS.md
Normal file
1096
docs/implplan/SPRINTS.md
Normal file
File diff suppressed because it is too large
Load Diff
208
docs/implplan/SPRINTS_PRIOR_20251019.md
Normal file
208
docs/implplan/SPRINTS_PRIOR_20251019.md
Normal file
@@ -0,0 +1,208 @@
|
||||
Closed sprint tasks archived from SPRINTS.md on 2025-10-19.
|
||||
|
||||
| Sprint | Theme | Tasks File Path | Status | Type of Specialist | Task ID | Task Description |
|
||||
| --- | --- | --- | --- | --- | --- | --- |
|
||||
| Sprint 1 | Stabilize In-Progress Foundations | src/Concelier/__Libraries/StellaOps.Concelier.Models/TASKS.md | DONE (2025-10-12) | Team Models & Merge Leads | FEEDMODELS-SCHEMA-01-001 | SemVer primitive range-style metadata<br>Instructions to work:<br>DONE Read ./AGENTS.md and src/Concelier/__Libraries/StellaOps.Concelier.Models/AGENTS.md. This task lays the groundwork—complete the SemVer helper updates before teammates pick up FEEDMODELS-SCHEMA-01-002/003 and FEEDMODELS-SCHEMA-02-900. Use ./src/FASTER_MODELING_AND_NORMALIZATION.md for the target rule structure. |
|
||||
| Sprint 1 | Stabilize In-Progress Foundations | src/Concelier/__Libraries/StellaOps.Concelier.Models/TASKS.md | DONE (2025-10-11) | Team Models & Merge Leads | FEEDMODELS-SCHEMA-01-002 | Provenance decision rationale field<br>Instructions to work:<br>AdvisoryProvenance now carries `decisionReason` and docs/tests were updated. Connectors and merge tasks should populate the field when applying precedence/freshness/tie-breaker logic; see src/Concelier/__Libraries/StellaOps.Concelier.Models/PROVENANCE_GUIDELINES.md for usage guidance. |
|
||||
| Sprint 1 | Stabilize In-Progress Foundations | src/Concelier/__Libraries/StellaOps.Concelier.Models/TASKS.md | DONE (2025-10-11) | Team Models & Merge Leads | FEEDMODELS-SCHEMA-01-003 | Normalized version rules collection<br>Instructions to work:<br>`AffectedPackage.NormalizedVersions` and supporting comparer/docs/tests shipped. Connector owners must emit rule arrays per ./src/FASTER_MODELING_AND_NORMALIZATION.md and report progress via FEEDMERGE-COORD-02-900 so merge/storage backfills can proceed. |
|
||||
| Sprint 1 | Stabilize In-Progress Foundations | src/Concelier/__Libraries/StellaOps.Concelier.Models/TASKS.md | DONE (2025-10-12) | Team Models & Merge Leads | FEEDMODELS-SCHEMA-02-900 | Range primitives for SemVer/EVR/NEVRA metadata<br>Instructions to work:<br>DONE Read ./AGENTS.md and src/Concelier/__Libraries/StellaOps.Concelier.Models/AGENTS.md before resuming this stalled effort. Confirm helpers align with the new `NormalizedVersions` representation so connectors finishing in Sprint 2 can emit consistent metadata. |
|
||||
| Sprint 1 | Stabilize In-Progress Foundations | src/Concelier/__Libraries/StellaOps.Concelier.Normalization/TASKS.md | DONE (2025-10-11) | Team Normalization & Storage Backbone | FEEDNORM-NORM-02-001 | SemVer normalized rule emitter<br>Shared `SemVerRangeRuleBuilder` now outputs primitives + normalized rules per `FASTER_MODELING_AND_NORMALIZATION.md`; CVE/GHSA connectors consuming the API have verified fixtures. |
|
||||
| Sprint 1 | Stabilize In-Progress Foundations | src/Concelier/__Libraries/StellaOps.Concelier.Storage.Mongo/TASKS.md | DONE (2025-10-11) | Team Normalization & Storage Backbone | FEEDSTORAGE-DATA-02-001 | Normalized range dual-write + backfill<br>AdvisoryStore dual-writes flattened `normalizedVersions` when `concelier.storage.enableSemVerStyle` is set; migration `20251011-semver-style-backfill` updates historical records and docs outline the rollout. |
|
||||
| Sprint 1 | Stabilize In-Progress Foundations | src/Concelier/__Libraries/StellaOps.Concelier.Storage.Mongo/TASKS.md | DONE (2025-10-11) | Team Normalization & Storage Backbone | FEEDSTORAGE-DATA-02-002 | Provenance decision reason persistence<br>Storage now persists `provenance.decisionReason` for advisories and merge events; tests cover round-trips. |
|
||||
| Sprint 1 | Stabilize In-Progress Foundations | src/Concelier/__Libraries/StellaOps.Concelier.Storage.Mongo/TASKS.md | DONE (2025-10-11) | Team Normalization & Storage Backbone | FEEDSTORAGE-DATA-02-003 | Normalized versions indexing<br>Bootstrapper seeds compound/sparse indexes for flattened normalized rules and `docs/dev/mongo_indices.md` documents query guidance. |
|
||||
| Sprint 1 | Stabilize In-Progress Foundations | src/Concelier/__Libraries/StellaOps.Concelier.Storage.Mongo/TASKS.md | DONE (2025-10-11) | Team Normalization & Storage Backbone | FEEDSTORAGE-TESTS-02-004 | Restore AdvisoryStore build after normalized versions refactor<br>Updated constructors/tests keep storage suites passing with the new feature flag defaults. |
|
||||
| Sprint 1 | Stabilize In-Progress Foundations | src/Concelier/StellaOps.Concelier.WebService/TASKS.md | DONE (2025-10-12) | Team WebService & Authority | FEEDWEB-ENGINE-01-002 | Plumb Authority client resilience options<br>WebService wires `authority.resilience.*` into `AddStellaOpsAuthClient` and adds binding coverage via `AuthorityClientResilienceOptionsAreBound`. |
|
||||
| Sprint 1 | Stabilize In-Progress Foundations | src/Concelier/StellaOps.Concelier.WebService/TASKS.md | DONE (2025-10-12) | Team WebService & Authority | FEEDWEB-DOCS-01-003 | Author ops guidance for resilience tuning<br>Install/runbooks document connected vs air-gapped resilience profiles and monitoring hooks. |
|
||||
| Sprint 1 | Stabilize In-Progress Foundations | src/Concelier/StellaOps.Concelier.WebService/TASKS.md | DONE (2025-10-12) | Team WebService & Authority | FEEDWEB-DOCS-01-004 | Document authority bypass logging patterns<br>Operator guides now call out `route/status/subject/clientId/scopes/bypass/remote` audit fields and SIEM triggers. |
|
||||
| Sprint 1 | Stabilize In-Progress Foundations | src/Concelier/StellaOps.Concelier.WebService/TASKS.md | DONE (2025-10-12) | Team WebService & Authority | FEEDWEB-DOCS-01-005 | Update Concelier operator guide for enforcement cutoff<br>Install guide reiterates the 2025-12-31 cutoff and links audit signals to the rollout checklist. |
|
||||
| Sprint 1 | Stabilize In-Progress Foundations | src/Authority/StellaOps.Authority/TASKS.md | DONE (2025-10-11) | Team WebService & Authority | SEC3.HOST | Rate limiter policy binding<br>Authority host now applies configuration-driven fixed windows to `/token`, `/authorize`, and `/internal/*`; integration tests assert 429 + `Retry-After` headers; docs/config samples refreshed for Docs guild diagrams. |
|
||||
| Sprint 1 | Stabilize In-Progress Foundations | src/Authority/StellaOps.Authority/TASKS.md | DONE (2025-10-11) | Team WebService & Authority | SEC3.BUILD | Authority rate-limiter follow-through<br>`Security.RateLimiting` now fronts token/authorize/internal limiters; Authority + Configuration matrices (`dotnet test src/Authority/StellaOps.Authority/StellaOps.Authority.sln`, `dotnet test src/__Libraries/__Tests/StellaOps.Configuration.Tests/StellaOps.Configuration.Tests.csproj`) passed on 2025-10-11; awaiting #authority-core broadcast. |
|
||||
| Sprint 1 | Stabilize In-Progress Foundations | src/Authority/StellaOps.Authority/TASKS.md | DONE (2025-10-14) | Team Authority Platform & Security Guild | AUTHCORE-BUILD-OPENIDDICT / AUTHCORE-STORAGE-DEVICE-TOKENS / AUTHCORE-BOOTSTRAP-INVITES | Address remaining Authority compile blockers (OpenIddict transaction shim, token device document, bootstrap invite cleanup) so `dotnet build src/Authority/StellaOps.Authority/StellaOps.Authority.sln` returns success. |
|
||||
| Sprint 1 | Stabilize In-Progress Foundations | src/Authority/StellaOps.Authority/StellaOps.Authority.Plugin.Standard/TASKS.md | DONE (2025-10-11) | Team WebService & Authority | PLG6.DOC | Plugin developer guide polish<br>Section 9 now documents rate limiter metadata, config keys, and lockout interplay; YAML samples updated alongside Authority config templates. |
|
||||
| Sprint 1 | Stabilize In-Progress Foundations | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.CertCc/TASKS.md | DONE (2025-10-11) | Team Connector Resumption – CERT/RedHat | FEEDCONN-CERTCC-02-001 | Fetch pipeline & state tracking<br>Summary planner now drives monthly/yearly VINCE fetches, persists pending summaries/notes, and hydrates VINCE detail queue with telemetry.<br>Team instructions: Read ./AGENTS.md and src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.CertCc/AGENTS.md. Coordinate daily with Models/Merge leads so new normalizedVersions output and provenance tags stay aligned with ./src/FASTER_MODELING_AND_NORMALIZATION.md. |
|
||||
| Sprint 1 | Stabilize In-Progress Foundations | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.CertCc/TASKS.md | DONE (2025-10-11) | Team Connector Resumption – CERT/RedHat | FEEDCONN-CERTCC-02-002 | VINCE note detail fetcher<br>Summary planner queues VINCE note detail endpoints, persists raw JSON with SHA/ETag metadata, and records retry/backoff metrics. |
|
||||
| Sprint 1 | Stabilize In-Progress Foundations | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.CertCc/TASKS.md | DONE (2025-10-11) | Team Connector Resumption – CERT/RedHat | FEEDCONN-CERTCC-02-003 | DTO & parser implementation<br>Added VINCE DTO aggregate, Markdown→text sanitizer, vendor/status/vulnerability parsers, and parser regression fixture. |
|
||||
| Sprint 1 | Stabilize In-Progress Foundations | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.CertCc/TASKS.md | DONE (2025-10-11) | Team Connector Resumption – CERT/RedHat | FEEDCONN-CERTCC-02-004 | Canonical mapping & range primitives<br>VINCE DTO aggregate flows through `CertCcMapper`, emitting vendor range primitives + normalized version rules that persist via `_advisoryStore`. |
|
||||
| Sprint 1 | Stabilize In-Progress Foundations | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.CertCc/TASKS.md | DONE (2025-10-12) | Team Connector Resumption – CERT/RedHat | FEEDCONN-CERTCC-02-005 | Deterministic fixtures/tests<br>Snapshot harness refreshed 2025-10-12; `certcc-*.snapshot.json` regenerated and regression suite green without UPDATE flag drift. |
|
||||
| Sprint 1 | Stabilize In-Progress Foundations | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.CertCc/TASKS.md | DONE (2025-10-12) | Team Connector Resumption – CERT/RedHat | FEEDCONN-CERTCC-02-006 | Telemetry & documentation<br>`CertCcDiagnostics` publishes summary/detail/parse/map metrics (meter `StellaOps.Concelier.Connector.CertCc`), README documents instruments, and log guidance captured for Ops on 2025-10-12. |
|
||||
| Sprint 1 | Stabilize In-Progress Foundations | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.CertCc/TASKS.md | DONE (2025-10-12) | Team Connector Resumption – CERT/RedHat | FEEDCONN-CERTCC-02-007 | Connector test harness remediation<br>Harness now wires `AddSourceCommon`, resets `FakeTimeProvider`, and passes canned-response regression run dated 2025-10-12. |
|
||||
| Sprint 1 | Stabilize In-Progress Foundations | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.CertCc/TASKS.md | DONE (2025-10-11) | Team Connector Resumption – CERT/RedHat | FEEDCONN-CERTCC-02-008 | Snapshot coverage handoff<br>Fixtures regenerated with normalized ranges + provenance fields on 2025-10-11; QA handoff notes published and merge backfill unblocked. |
|
||||
| Sprint 1 | Stabilize In-Progress Foundations | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.CertCc/TASKS.md | DONE (2025-10-12) | Team Connector Resumption – CERT/RedHat | FEEDCONN-CERTCC-02-012 | Schema sync & snapshot regen follow-up<br>Fixtures regenerated with normalizedVersions + provenance decision reasons; handoff notes updated for Merge backfill 2025-10-12. |
|
||||
| Sprint 1 | Stabilize In-Progress Foundations | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.CertCc/TASKS.md | DONE (2025-10-11) | Team Connector Resumption – CERT/RedHat | FEEDCONN-CERTCC-02-009 | Detail/map reintegration plan<br>Staged reintegration plan published in `src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.CertCc/FEEDCONN-CERTCC-02-009_PLAN.md`; coordinates enablement with FEEDCONN-CERTCC-02-004. |
|
||||
| Sprint 1 | Stabilize In-Progress Foundations | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.CertCc/TASKS.md | DONE (2025-10-12) | Team Connector Resumption – CERT/RedHat | FEEDCONN-CERTCC-02-010 | Partial-detail graceful degradation<br>Detail fetch now tolerates 404/403/410 responses and regression tests cover mixed endpoint availability. |
|
||||
| Sprint 1 | Stabilize In-Progress Foundations | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Distro.RedHat/TASKS.md | DONE (2025-10-11) | Team Connector Resumption – CERT/RedHat | FEEDCONN-REDHAT-02-001 | Fixture validation sweep<br>Instructions to work:<br>Fixtures regenerated post-model-helper rollout; provenance ordering and normalizedVersions scaffolding verified via tests. Conflict resolver deltas logged in src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Distro.RedHat/CONFLICT_RESOLVER_NOTES.md for Sprint 3 consumers. |
|
||||
| Sprint 1 | Stabilize In-Progress Foundations | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Vndr.Apple/TASKS.md | DONE (2025-10-12) | Team Vendor Apple Specialists | FEEDCONN-APPLE-02-001 | Canonical mapping & range primitives<br>Mapper emits SemVer rules (`scheme=apple:*`); fixtures regenerated with trimmed references + new RSR coverage, update tooling finalized. |
|
||||
| Sprint 1 | Stabilize In-Progress Foundations | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Vndr.Apple/TASKS.md | DONE (2025-10-11) | Team Vendor Apple Specialists | FEEDCONN-APPLE-02-002 | Deterministic fixtures/tests<br>Sanitized live fixtures + regression snapshots wired into tests; normalized rule coverage asserted. |
|
||||
| Sprint 1 | Stabilize In-Progress Foundations | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Vndr.Apple/TASKS.md | DONE (2025-10-11) | Team Vendor Apple Specialists | FEEDCONN-APPLE-02-003 | Telemetry & documentation<br>Apple meter metrics wired into Concelier WebService OpenTelemetry configuration; README and fixtures document normalizedVersions coverage. |
|
||||
| Sprint 1 | Stabilize In-Progress Foundations | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Vndr.Apple/TASKS.md | DONE (2025-10-12) | Team Vendor Apple Specialists | FEEDCONN-APPLE-02-004 | Live HTML regression sweep<br>Sanitised HT125326/HT125328/HT106355/HT214108/HT215500 fixtures recorded and regression tests green on 2025-10-12. |
|
||||
| Sprint 1 | Stabilize In-Progress Foundations | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Vndr.Apple/TASKS.md | DONE (2025-10-11) | Team Vendor Apple Specialists | FEEDCONN-APPLE-02-005 | Fixture regeneration tooling<br>`UPDATE_APPLE_FIXTURES=1` flow fetches & rewrites fixtures; README documents usage.<br>Instructions to work:<br>DONE Read ./AGENTS.md and src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Vndr.Apple/AGENTS.md. Resume stalled tasks, ensuring normalizedVersions output and fixtures align with ./src/FASTER_MODELING_AND_NORMALIZATION.md before handing data to the conflict sprint. |
|
||||
| Sprint 1 | Stabilize In-Progress Foundations | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Ghsa/TASKS.md | DONE (2025-10-12) | Team Connector Normalized Versions Rollout | FEEDCONN-GHSA-02-001 | GHSA normalized versions & provenance<br>Team instructions: Read ./AGENTS.md and each module's AGENTS file. Adopt the `NormalizedVersions` array emitted by the models sprint, wiring provenance `decisionReason` where merge overrides occur. Follow ./src/FASTER_MODELING_AND_NORMALIZATION.md; report via src/Concelier/__Libraries/StellaOps.Concelier.Merge/TASKS.md (FEEDMERGE-COORD-02-900). Progress 2025-10-11: GHSA/OSV emit normalized arrays with refreshed fixtures; CVE mapper now surfaces SemVer normalized ranges; NVD/KEV adoption pending; outstanding follow-ups include FEEDSTORAGE-DATA-02-001, FEEDMERGE-ENGINE-02-002, and rolling `tools/FixtureUpdater` updates across connectors. |
|
||||
| Sprint 1 | Stabilize In-Progress Foundations | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Osv/TASKS.md | DONE (2025-10-12) | Team Connector Normalized Versions Rollout | FEEDCONN-OSV-02-003 | OSV normalized versions & freshness |
|
||||
| Sprint 1 | Stabilize In-Progress Foundations | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Nvd/TASKS.md | DONE (2025-10-12) | Team Connector Normalized Versions Rollout | FEEDCONN-NVD-02-002 | NVD normalized versions & timestamps |
|
||||
| Sprint 1 | Stabilize In-Progress Foundations | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Cve/TASKS.md | DONE (2025-10-12) | Team Connector Normalized Versions Rollout | FEEDCONN-CVE-02-003 | CVE normalized versions uplift |
|
||||
| Sprint 1 | Stabilize In-Progress Foundations | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Kev/TASKS.md | DONE (2025-10-12) | Team Connector Normalized Versions Rollout | FEEDCONN-KEV-02-003 | KEV normalized versions propagation |
|
||||
| Sprint 1 | Stabilize In-Progress Foundations | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Osv/TASKS.md | DONE (2025-10-12) | Team Connector Normalized Versions Rollout | FEEDCONN-OSV-04-003 | OSV parity fixture refresh |
|
||||
| Sprint 1 | Stabilize In-Progress Foundations | src/Concelier/StellaOps.Concelier.WebService/TASKS.md | DONE (2025-10-10) | Team WebService & Authority | FEEDWEB-DOCS-01-001 | Document authority toggle & scope requirements<br>Quickstart carries toggle/scope guidance pending docs guild review (no change this sprint). |
|
||||
| Sprint 1 | Stabilize In-Progress Foundations | src/Concelier/StellaOps.Concelier.WebService/TASKS.md | DONE (2025-10-12) | Team WebService & Authority | FEEDWEB-DOCS-01-003 | Author ops guidance for resilience tuning<br>Operator docs now outline connected vs air-gapped resilience profiles and monitoring cues. |
|
||||
| Sprint 1 | Stabilize In-Progress Foundations | src/Concelier/StellaOps.Concelier.WebService/TASKS.md | DONE (2025-10-12) | Team WebService & Authority | FEEDWEB-DOCS-01-004 | Document authority bypass logging patterns<br>Audit logging guidance highlights `route/status/subject/clientId/scopes/bypass/remote` fields and SIEM alerts. |
|
||||
| Sprint 1 | Stabilize In-Progress Foundations | src/Concelier/StellaOps.Concelier.WebService/TASKS.md | DONE (2025-10-12) | Team WebService & Authority | FEEDWEB-DOCS-01-005 | Update Concelier operator guide for enforcement cutoff<br>Install guide reiterates the 2025-12-31 cutoff and ties audit signals to rollout checks. |
|
||||
| Sprint 1 | Stabilize In-Progress Foundations | src/Concelier/StellaOps.Concelier.WebService/TASKS.md | DONE (2025-10-11) | Team WebService & Authority | FEEDWEB-OPS-01-006 | Rename plugin drop directory to namespaced path<br>Build outputs, tests, and docs now target `StellaOps.Concelier.PluginBinaries`/`StellaOps.Authority.PluginBinaries`. |
|
||||
| Sprint 1 | Stabilize In-Progress Foundations | src/Concelier/StellaOps.Concelier.WebService/TASKS.md | DONE (2025-10-11) | Team WebService & Authority | FEEDWEB-OPS-01-007 | Authority resilience adoption<br>Deployment docs and CLI notes explain the LIB5 resilience knobs for rollout.<br>Instructions to work:<br>DONE Read ./AGENTS.md and src/Concelier/StellaOps.Concelier.WebService/AGENTS.md. These items were mid-flight; resume implementation ensuring docs/operators receive timely updates. |
|
||||
| Sprint 1 | Stabilize In-Progress Foundations | src/Authority/StellaOps.Authority/TASKS.md | DONE (2025-10-11) | Team Authority Platform & Security Guild | AUTHCORE-ENGINE-01-001 | CORE8.RL — Rate limiter plumbing validated; integration tests green and docs handoff recorded for middleware ordering + Retry-After headers (see `docs/dev/authority-rate-limit-tuning-outline.md` for continuing guidance). |
|
||||
| Sprint 1 | Stabilize In-Progress Foundations | src/__Libraries/StellaOps.Cryptography/TASKS.md | DONE (2025-10-11) | Team Authority Platform & Security Guild | AUTHCRYPTO-ENGINE-01-001 | SEC3.A — Shared metadata resolver confirmed via host test run; SEC3.B now unblocked for tuning guidance (outline captured in `docs/dev/authority-rate-limit-tuning-outline.md`). |
|
||||
| Sprint 1 | Stabilize In-Progress Foundations | src/__Libraries/StellaOps.Cryptography/TASKS.md | DONE (2025-10-13) | Team Authority Platform & Security Guild | AUTHSEC-DOCS-01-002 | SEC3.B — Published `docs/security/rate-limits.md` with tuning matrix, alert thresholds, and lockout interplay guidance; Docs guild can lift copy into plugin guide. |
|
||||
| Sprint 1 | Stabilize In-Progress Foundations | src/__Libraries/StellaOps.Cryptography/TASKS.md | DONE (2025-10-14) | Team Authority Platform & Security Guild | AUTHSEC-CRYPTO-02-001 | SEC5.B1 — Introduce libsodium signing provider and parity tests to unblock CLI verification enhancements. |
|
||||
| Sprint 1 | Bootstrap & Replay Hardening | src/__Libraries/StellaOps.Cryptography/TASKS.md | DONE (2025-10-14) | Security Guild | AUTHSEC-CRYPTO-02-004 | SEC5.D/E — Finish bootstrap invite lifecycle (API/store/cleanup) and token device heuristics; build currently red due to pending handler integration. |
|
||||
| Sprint 1 | Developer Tooling | src/Cli/StellaOps.Cli/TASKS.md | DONE (2025-10-15) | DevEx/CLI | AUTHCLI-DIAG-01-001 | Surface password policy diagnostics in CLI startup/output so operators see weakened overrides immediately.<br>CLI now loads Authority plug-ins at startup, logs weakened password policies (length/complexity), and regression coverage lives in `StellaOps.Cli.Tests/Services/AuthorityDiagnosticsReporterTests`. |
|
||||
| Sprint 1 | Stabilize In-Progress Foundations | src/Authority/StellaOps.Authority/StellaOps.Authority.Plugin.Standard/TASKS.md | DONE (2025-10-11) | Team Authority Platform & Security Guild | AUTHPLUG-DOCS-01-001 | PLG6.DOC — Developer guide copy + diagrams merged 2025-10-11; limiter guidance incorporated and handed to Docs guild for asset export. |
|
||||
| Sprint 2 | Connector & Data Implementation Wave | src/Concelier/__Libraries/StellaOps.Concelier.Normalization/TASKS.md | DONE (2025-10-12) | Team Normalization & Storage Backbone | FEEDNORM-NORM-02-001 | SemVer normalized rule emitter<br>`SemVerRangeRuleBuilder` shipped 2025-10-12 with comparator/`||` support and fixtures aligning to `FASTER_MODELING_AND_NORMALIZATION.md`. |
|
||||
| Sprint 2 | Connector & Data Implementation Wave | src/Concelier/__Libraries/StellaOps.Concelier.Storage.Mongo/TASKS.md | DONE (2025-10-11) | Team Normalization & Storage Backbone | FEEDSTORAGE-DATA-02-001 | Normalized range dual-write + backfill |
|
||||
| Sprint 2 | Connector & Data Implementation Wave | src/Concelier/__Libraries/StellaOps.Concelier.Storage.Mongo/TASKS.md | DONE (2025-10-11) | Team Normalization & Storage Backbone | FEEDSTORAGE-DATA-02-002 | Provenance decision reason persistence |
|
||||
| Sprint 2 | Connector & Data Implementation Wave | src/Concelier/__Libraries/StellaOps.Concelier.Storage.Mongo/TASKS.md | DONE (2025-10-11) | Team Normalization & Storage Backbone | FEEDSTORAGE-DATA-02-003 | Normalized versions indexing<br>Indexes seeded + docs updated 2025-10-11 to cover flattened normalized rules for connector adoption. |
|
||||
| Sprint 2 | Connector & Data Implementation Wave | src/Concelier/__Libraries/StellaOps.Concelier.Merge/TASKS.md | DONE (2025-10-11) | Team Normalization & Storage Backbone | FEEDMERGE-ENGINE-02-002 | Normalized versions union & dedupe<br>Affected package resolver unions/dedupes normalized rules, stamps merge provenance with `decisionReason`, and tests cover the rollout. |
|
||||
| Sprint 2 | Connector & Data Implementation Wave | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Ghsa/TASKS.md | DONE (2025-10-11) | Team Connector Expansion – GHSA/NVD/OSV | FEEDCONN-GHSA-02-001 | GHSA normalized versions & provenance |
|
||||
| Sprint 2 | Connector & Data Implementation Wave | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Ghsa/TASKS.md | DONE (2025-10-11) | Team Connector Expansion – GHSA/NVD/OSV | FEEDCONN-GHSA-02-004 | GHSA credits & ecosystem severity mapping |
|
||||
| Sprint 2 | Connector & Data Implementation Wave | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Ghsa/TASKS.md | DONE (2025-10-12) | Team Connector Expansion – GHSA/NVD/OSV | FEEDCONN-GHSA-02-005 | GitHub quota monitoring & retries |
|
||||
| Sprint 2 | Connector & Data Implementation Wave | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Ghsa/TASKS.md | DONE (2025-10-12) | Team Connector Expansion – GHSA/NVD/OSV | FEEDCONN-GHSA-02-006 | Production credential & scheduler rollout |
|
||||
| Sprint 2 | Connector & Data Implementation Wave | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Ghsa/TASKS.md | DONE (2025-10-12) | Team Connector Expansion – GHSA/NVD/OSV | FEEDCONN-GHSA-02-007 | Credit parity regression fixtures |
|
||||
| Sprint 2 | Connector & Data Implementation Wave | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Nvd/TASKS.md | DONE (2025-10-11) | Team Connector Expansion – GHSA/NVD/OSV | FEEDCONN-NVD-02-002 | NVD normalized versions & timestamps |
|
||||
| Sprint 2 | Connector & Data Implementation Wave | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Nvd/TASKS.md | DONE (2025-10-11) | Team Connector Expansion – GHSA/NVD/OSV | FEEDCONN-NVD-02-004 | NVD CVSS & CWE precedence payloads |
|
||||
| Sprint 2 | Connector & Data Implementation Wave | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Nvd/TASKS.md | DONE (2025-10-12) | Team Connector Expansion – GHSA/NVD/OSV | FEEDCONN-NVD-02-005 | NVD merge/export parity regression |
|
||||
| Sprint 2 | Connector & Data Implementation Wave | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Osv/TASKS.md | DONE (2025-10-11) | Team Connector Expansion – GHSA/NVD/OSV | FEEDCONN-OSV-02-003 | OSV normalized versions & freshness |
|
||||
| Sprint 2 | Connector & Data Implementation Wave | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Osv/TASKS.md | DONE (2025-10-11) | Team Connector Expansion – GHSA/NVD/OSV | FEEDCONN-OSV-02-004 | OSV references & credits alignment |
|
||||
| Sprint 2 | Connector & Data Implementation Wave | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Osv/TASKS.md | DONE (2025-10-12) | Team Connector Expansion – GHSA/NVD/OSV | FEEDCONN-OSV-02-005 | Fixture updater workflow<br>Resolved 2025-10-12: OSV mapper now derives canonical PURLs for Go + scoped npm packages when raw payloads omit `purl`; conflict fixtures unchanged for invalid npm names. Verified via `dotnet test src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Osv.Tests`, `src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Ghsa.Tests`, `src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Nvd.Tests`, and backbone normalization/storage suites. |
|
||||
| Sprint 2 | Connector & Data Implementation Wave | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Acsc/TASKS.md | DONE (2025-10-12) | Team Connector Expansion – Regional & Vendor Feeds | FEEDCONN-ACSC-02-001 … 02-008 | Fetch→parse→map pipeline, fixtures, diagnostics, and README finished 2025-10-12; downstream export parity captured via FEEDEXPORT-JSON-04-001 / FEEDEXPORT-TRIVY-04-001 (completed). |
|
||||
| Sprint 2 | Connector & Data Implementation Wave | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Cccs/TASKS.md | DONE (2025-10-16) | Team Connector Expansion – Regional & Vendor Feeds | FEEDCONN-CCCS-02-001 … 02-008 | Observability meter, historical harvest plan, and DOM sanitizer refinements wrapped; ops notes live under `docs/ops/concelier-cccs-operations.md` with fixtures validating EN/FR list handling. |
|
||||
| Sprint 2 | Connector & Data Implementation Wave | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.CertBund/TASKS.md | DONE (2025-10-15) | Team Connector Expansion – Regional & Vendor Feeds | FEEDCONN-CERTBUND-02-001 … 02-008 | Telemetry/docs (02-006) and history/locale sweep (02-007) completed alongside pipeline; runbook `docs/ops/concelier-certbund-operations.md` captures locale guidance and offline packaging. |
|
||||
| Sprint 2 | Connector & Data Implementation Wave | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Kisa/TASKS.md | DONE (2025-10-14) | Team Connector Expansion – Regional & Vendor Feeds | FEEDCONN-KISA-02-001 … 02-007 | Connector, tests, and telemetry/docs (02-006) finalized; localisation notes in `docs/dev/kisa_connector_notes.md` complete rollout. |
|
||||
| Sprint 2 | Connector & Data Implementation Wave | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Ru.Bdu/TASKS.md | DONE (2025-10-14) | Team Connector Expansion – Regional & Vendor Feeds | FEEDCONN-RUBDU-02-001 … 02-008 | Fetch/parser/mapper refinements, regression fixtures, telemetry/docs, access options, and trusted root packaging all landed; README documents offline access strategy. |
|
||||
| Sprint 2 | Connector & Data Implementation Wave | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Ru.Nkcki/TASKS.md | DONE (2025-10-13) | Team Connector Expansion – Regional & Vendor Feeds | FEEDCONN-NKCKI-02-001 … 02-008 | Listing fetch, parser, mapper, fixtures, telemetry/docs, and archive plan finished; Mongo2Go/libcrypto dependency resolved via bundled OpenSSL noted in ops guide. |
|
||||
| Sprint 2 | Connector & Data Implementation Wave | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Ics.Cisa/TASKS.md | DONE (2025-10-16) | Team Connector Expansion – Regional & Vendor Feeds | FEEDCONN-ICSCISA-02-001 … 02-011 | Feed parser attachment fixes, SemVer exact values, regression suites, telemetry/docs updates, and handover complete; ops runbook now details attachment verification + proxy usage. |
|
||||
| Sprint 2 | Connector & Data Implementation Wave | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Vndr.Cisco/TASKS.md | DONE (2025-10-14) | Team Connector Expansion – Regional & Vendor Feeds | FEEDCONN-CISCO-02-001 … 02-007 | OAuth fetch pipeline, DTO/mapping, tests, and telemetry/docs shipped; monitoring/export integration follow-ups recorded in Ops docs and exporter backlog (completed). |
|
||||
| Sprint 2 | Connector & Data Implementation Wave | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Vndr.Msrc/TASKS.md | DONE (2025-10-15) | Team Connector Expansion – Regional & Vendor Feeds | FEEDCONN-MSRC-02-001 … 02-008 | Azure AD onboarding (02-008) unblocked fetch/parse/map pipeline; fixtures, telemetry/docs, and Offline Kit guidance published in `docs/ops/concelier-msrc-operations.md`. |
|
||||
| Sprint 2 | Connector & Data Implementation Wave | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Cve/TASKS.md | DONE (2025-10-15) | Team Connector Support & Monitoring | FEEDCONN-CVE-02-001 … 02-002 | CVE data-source selection, fetch pipeline, and docs landed 2025-10-10. 2025-10-15: smoke verified using the seeded mirror fallback; connector now logs a warning and pulls from `seed-data/cve/` until live CVE Services credentials arrive. |
|
||||
| Sprint 2 | Connector & Data Implementation Wave | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Kev/TASKS.md | DONE (2025-10-12) | Team Connector Support & Monitoring | FEEDCONN-KEV-02-001 … 02-002 | KEV catalog ingestion, fixtures, telemetry, and schema validation completed 2025-10-12; ops dashboard published. |
|
||||
| Sprint 2 | Connector & Data Implementation Wave | docs/TASKS.md | DONE (2025-10-11) | Team Docs & Knowledge Base | FEEDDOCS-DOCS-01-001 | Canonical schema docs refresh<br>Updated canonical schema + provenance guides with SemVer style, normalized version rules, decision reason change log, and migration notes. |
|
||||
| Sprint 2 | Connector & Data Implementation Wave | docs/TASKS.md | DONE (2025-10-11) | Team Docs & Knowledge Base | FEEDDOCS-DOCS-02-001 | Concelier-SemVer Playbook<br>Published merge playbook covering mapper patterns, dedupe flow, indexes, and rollout checklist. |
|
||||
| Sprint 2 | Connector & Data Implementation Wave | docs/TASKS.md | DONE (2025-10-11) | Team Docs & Knowledge Base | FEEDDOCS-DOCS-02-002 | Normalized versions query guide<br>Delivered Mongo index/query addendum with `$unwind` recipes, dedupe checks, and operational checklist.<br>Instructions to work:<br>DONE Read ./AGENTS.md and docs/AGENTS.md. Document every schema/index/query change produced in Sprint 1-2 leveraging ./src/FASTER_MODELING_AND_NORMALIZATION.md. |
|
||||
| Sprint 3 | Conflict Resolution Integration & Communications | src/Concelier/__Libraries/StellaOps.Concelier.Core/TASKS.md | DONE (2025-10-11) | Team Core Engine & Storage Analytics | FEEDCORE-ENGINE-03-001 | Canonical merger implementation<br>`CanonicalMerger` ships with freshness/tie-breaker logic, provenance, and unit coverage feeding Merge. |
|
||||
| Sprint 3 | Conflict Resolution Integration & Communications | src/Concelier/__Libraries/StellaOps.Concelier.Core/TASKS.md | DONE (2025-10-11) | Team Core Engine & Storage Analytics | FEEDCORE-ENGINE-03-002 | Field precedence and tie-breaker map<br>Field precedence tables and tie-breaker metrics wired into the canonical merge flow; docs/tests updated.<br>Instructions to work:<br>Read ./AGENTS.md and core AGENTS. Implement the conflict resolver exactly as specified in ./src/DEDUP_CONFLICTS_RESOLUTION_ALGO.md, coordinating with Merge and Storage teammates. |
|
||||
| Sprint 3 | Conflict Resolution Integration & Communications | src/Concelier/__Libraries/StellaOps.Concelier.Storage.Mongo/TASKS.md | DONE (2025-10-11) | Team Core Engine & Storage Analytics | FEEDSTORAGE-DATA-03-001 | Merge event provenance audit prep<br>Merge events now persist `fieldDecisions` and analytics-ready provenance snapshots. |
|
||||
| Sprint 3 | Conflict Resolution Integration & Communications | src/Concelier/__Libraries/StellaOps.Concelier.Storage.Mongo/TASKS.md | DONE (2025-10-11) | Team Core Engine & Storage Analytics | FEEDSTORAGE-DATA-02-001 | Normalized range dual-write + backfill<br>Dual-write/backfill flag delivered; migration + options validated in tests. |
|
||||
| Sprint 3 | Conflict Resolution Integration & Communications | src/Concelier/__Libraries/StellaOps.Concelier.Storage.Mongo/TASKS.md | DONE (2025-10-11) | Team Core Engine & Storage Analytics | FEEDSTORAGE-TESTS-02-004 | Restore AdvisoryStore build after normalized versions refactor<br>Storage tests adjusted for normalized versions/decision reasons.<br>Instructions to work:<br>Read ./AGENTS.md and storage AGENTS. Extend merge events with decision reasons and analytics views to support the conflict rules, and deliver the dual-write/backfill for `NormalizedVersions` + `decisionReason` so connectors can roll out safely. |
|
||||
| Sprint 3 | Conflict Resolution Integration & Communications | src/Concelier/__Libraries/StellaOps.Concelier.Merge/TASKS.md | DONE (2025-10-11) | Team Merge & QA Enforcement | FEEDMERGE-ENGINE-04-001 | GHSA/NVD/OSV conflict rules<br>Merge pipeline consumes `CanonicalMerger` output prior to precedence merge. |
|
||||
| Sprint 3 | Conflict Resolution Integration & Communications | src/Concelier/__Libraries/StellaOps.Concelier.Merge/TASKS.md | DONE (2025-10-11) | Team Merge & QA Enforcement | FEEDMERGE-ENGINE-04-002 | Override metrics instrumentation<br>Merge events capture per-field decisions; counters/logs align with conflict rules. |
|
||||
| Sprint 3 | Conflict Resolution Integration & Communications | src/Concelier/__Libraries/StellaOps.Concelier.Merge/TASKS.md | DONE (2025-10-11) | Team Merge & QA Enforcement | FEEDMERGE-ENGINE-04-003 | Reference & credit union pipeline<br>Canonical merge preserves unions with updated tests. |
|
||||
| Sprint 3 | Conflict Resolution Integration & Communications | src/Concelier/__Libraries/StellaOps.Concelier.Merge/TASKS.md | DONE (2025-10-11) | Team Merge & QA Enforcement | FEEDMERGE-QA-04-001 | End-to-end conflict regression suite<br>Added regression tests (`AdvisoryMergeServiceTests`) covering canonical + precedence flow.<br>Instructions to work:<br>Read ./AGENTS.md and merge AGENTS. Integrate the canonical merger, instrument metrics, and deliver comprehensive regression tests following ./src/DEDUP_CONFLICTS_RESOLUTION_ALGO.md. |
|
||||
| Sprint 3 | Conflict Resolution Integration & Communications | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Ghsa/TASKS.md | DONE (2025-10-12) | Team Connector Regression Fixtures | FEEDCONN-GHSA-04-002 | GHSA conflict regression fixtures |
|
||||
| Sprint 3 | Conflict Resolution Integration & Communications | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Nvd/TASKS.md | DONE (2025-10-12) | Team Connector Regression Fixtures | FEEDCONN-NVD-04-002 | NVD conflict regression fixtures |
|
||||
| Sprint 3 | Conflict Resolution Integration & Communications | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Osv/TASKS.md | DONE (2025-10-12) | Team Connector Regression Fixtures | FEEDCONN-OSV-04-002 | OSV conflict regression fixtures<br>Instructions to work:<br>Read ./AGENTS.md and module AGENTS. Produce fixture triples supporting the precedence/tie-breaker paths defined in ./src/DEDUP_CONFLICTS_RESOLUTION_ALGO.md and hand them to Merge QA. |
|
||||
| Sprint 3 | Conflict Resolution Integration & Communications | docs/TASKS.md | DONE (2025-10-11) | Team Documentation Guild – Conflict Guidance | FEEDDOCS-DOCS-05-001 | Concelier Conflict Rules<br>Runbook published at `docs/ops/concelier-conflict-resolution.md`; metrics/log guidance aligned with Sprint 3 merge counters. |
|
||||
| Sprint 3 | Conflict Resolution Integration & Communications | docs/TASKS.md | DONE (2025-10-16) | Team Documentation Guild – Conflict Guidance | FEEDDOCS-DOCS-05-002 | Conflict runbook ops rollout<br>Ops review completed, alert thresholds applied, and change log appended in `docs/ops/concelier-conflict-resolution.md`; task closed after connector signals verified. |
|
||||
| Sprint 4 | Schema Parity & Freshness Alignment | src/Concelier/__Libraries/StellaOps.Concelier.Models/TASKS.md | DONE (2025-10-15) | Team Models & Merge Leads | FEEDMODELS-SCHEMA-04-001 | Advisory schema parity (description/CWE/canonical metric)<br>Extend `Advisory` and related records with description text, CWE collection, and canonical metric pointer; refresh validation + serializer determinism tests. |
|
||||
| Sprint 4 | Schema Parity & Freshness Alignment | src/Concelier/__Libraries/StellaOps.Concelier.Core/TASKS.md | DONE (2025-10-15) | Team Core Engine & Storage Analytics | FEEDCORE-ENGINE-04-003 | Canonical merger parity for new fields<br>Teach `CanonicalMerger` to populate description, CWEResults, and canonical metric pointer with provenance + regression coverage. |
|
||||
| Sprint 4 | Schema Parity & Freshness Alignment | src/Concelier/__Libraries/StellaOps.Concelier.Core/TASKS.md | DONE (2025-10-15) | Team Core Engine & Storage Analytics | FEEDCORE-ENGINE-04-004 | Reference normalization & freshness instrumentation cleanup<br>Implement URL normalization for reference dedupe, align freshness-sensitive instrumentation, and add analytics tests. |
|
||||
| Sprint 4 | Schema Parity & Freshness Alignment | src/Concelier/__Libraries/StellaOps.Concelier.Merge/TASKS.md | DONE (2025-10-15) | Team Merge & QA Enforcement | FEEDMERGE-ENGINE-04-004 | Merge pipeline parity for new advisory fields<br>Ensure merge service + merge events surface description/CWE/canonical metric decisions with updated metrics/tests. |
|
||||
| Sprint 4 | Schema Parity & Freshness Alignment | src/Concelier/__Libraries/StellaOps.Concelier.Merge/TASKS.md | DONE (2025-10-15) | Team Merge & QA Enforcement | FEEDMERGE-ENGINE-04-005 | Connector coordination for new advisory fields<br>GHSA/NVD/OSV connectors now ship description, CWE, and canonical metric data with refreshed fixtures; merge coordination log updated and exporters notified. |
|
||||
| Sprint 4 | Schema Parity & Freshness Alignment | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Exporter.Json/TASKS.md | DONE (2025-10-15) | Team Exporters – JSON | FEEDEXPORT-JSON-04-001 | Surface new advisory fields in JSON exporter<br>Update schemas/offline bundle + fixtures once model/core parity lands.<br>2025-10-15: `dotnet test src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Exporter.Json.Tests` validated canonical metric/CWE emission. |
|
||||
| Sprint 4 | Schema Parity & Freshness Alignment | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Exporter.TrivyDb/TASKS.md | DONE (2025-10-15) | Team Exporters – Trivy DB | FEEDEXPORT-TRIVY-04-001 | Propagate new advisory fields into Trivy DB package<br>Extend Bolt builder, metadata, and regression tests for the expanded schema.<br>2025-10-15: `dotnet test src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Exporter.TrivyDb.Tests` confirmed canonical metric/CWE propagation. |
|
||||
| Sprint 4 | Schema Parity & Freshness Alignment | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Ghsa/TASKS.md | DONE (2025-10-16) | Team Connector Regression Fixtures | FEEDCONN-GHSA-04-004 | Harden CVSS fallback so canonical metric ids persist when GitHub omits vectors; extend fixtures and document severity precedence hand-off to Merge. |
|
||||
| Sprint 4 | Schema Parity & Freshness Alignment | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Osv/TASKS.md | DONE (2025-10-16) | Team Connector Expansion – GHSA/NVD/OSV | FEEDCONN-OSV-04-005 | Map OSV advisories lacking CVSS vectors to canonical metric ids/notes and document CWE provenance quirks; schedule parity fixture updates. |
|
||||
| Sprint 5 | Excititor Core Foundations | src/Excititor/__Libraries/StellaOps.Excititor.Core/TASKS.md | DONE (2025-10-15) | Team Excititor Core & Policy | EXCITITOR-CORE-01-001 | Stand up canonical VEX claim/consensus records with deterministic serializers so Storage/Exports share a stable contract. |
|
||||
| Sprint 5 | Excititor Core Foundations | src/Excititor/__Libraries/StellaOps.Excititor.Core/TASKS.md | DONE (2025-10-15) | Team Excititor Core & Policy | EXCITITOR-CORE-01-002 | Implement trust-weighted consensus resolver with baseline policy weights, justification gates, telemetry output, and majority/tie handling. |
|
||||
| Sprint 5 | Excititor Core Foundations | src/Excititor/__Libraries/StellaOps.Excititor.Core/TASKS.md | DONE (2025-10-15) | Team Excititor Core & Policy | EXCITITOR-CORE-01-003 | Publish shared connector/exporter/attestation abstractions and deterministic query signature utilities for cache/attestation workflows. |
|
||||
| Sprint 5 | Excititor Core Foundations | src/Excititor/__Libraries/StellaOps.Excititor.Policy/TASKS.md | DONE (2025-10-15) | Team Excititor Policy | EXCITITOR-POLICY-01-001 | Established policy options & snapshot provider covering baseline weights/overrides. |
|
||||
| Sprint 5 | Excititor Core Foundations | src/Excititor/__Libraries/StellaOps.Excititor.Policy/TASKS.md | DONE (2025-10-15) | Team Excititor Policy | EXCITITOR-POLICY-01-002 | Policy evaluator now feeds consensus resolver with immutable snapshots. |
|
||||
| Sprint 5 | Excititor Core Foundations | src/Excititor/__Libraries/StellaOps.Excititor.Policy/TASKS.md | DONE (2025-10-16) | Team Excititor Policy | EXCITITOR-POLICY-01-003 | Author policy diagnostics, CLI/WebService surfacing, and documentation updates. |
|
||||
| Sprint 5 | Excititor Core Foundations | src/Excititor/__Libraries/StellaOps.Excititor.Policy/TASKS.md | DONE (2025-10-16) | Team Excititor Policy | EXCITITOR-POLICY-01-004 | Implement YAML/JSON schema validation and deterministic diagnostics for operator bundles. |
|
||||
| Sprint 5 | Excititor Core Foundations | src/Excititor/__Libraries/StellaOps.Excititor.Policy/TASKS.md | DONE (2025-10-16) | Team Excititor Policy | EXCITITOR-POLICY-01-005 | Add policy change tracking, snapshot digests, and telemetry/logging hooks. |
|
||||
| Sprint 5 | Excititor Core Foundations | src/Excititor/__Libraries/StellaOps.Excititor.Storage.Mongo/TASKS.md | DONE (2025-10-15) | Team Excititor Storage | EXCITITOR-STORAGE-01-001 | Mongo mapping registry plus raw/export entities and DI extensions in place. |
|
||||
| Sprint 5 | Excititor Core Foundations | src/Excititor/__Libraries/StellaOps.Excititor.Storage.Mongo/TASKS.md | DONE (2025-10-16) | Team Excititor Storage | EXCITITOR-STORAGE-01-004 | Build provider/consensus/cache class maps and related collections. |
|
||||
| Sprint 5 | Excititor Core Foundations | src/Excititor/__Libraries/StellaOps.Excititor.Export/TASKS.md | DONE (2025-10-15) | Team Excititor Export | EXCITITOR-EXPORT-01-001 | Export engine delivers cache lookup, manifest creation, and policy integration. |
|
||||
| Sprint 5 | Excititor Core Foundations | src/Excititor/__Libraries/StellaOps.Excititor.Export/TASKS.md | DONE (2025-10-17) | Team Excititor Export | EXCITITOR-EXPORT-01-004 | Connect export engine to attestation client and persist Rekor metadata. |
|
||||
| Sprint 5 | Excititor Core Foundations | src/Excititor/__Libraries/StellaOps.Excititor.Attestation/TASKS.md | DONE (2025-10-16) | Team Excititor Attestation | EXCITITOR-ATTEST-01-001 | Implement in-toto predicate + DSSE builder providing envelopes for export attestation. |
|
||||
| Sprint 5 | Excititor Core Foundations | src/Excititor/__Libraries/StellaOps.Excititor.Connectors.Abstractions/TASKS.md | DONE (2025-10-17) | Team Excititor Connectors | EXCITITOR-CONN-ABS-01-001 | Deliver shared connector context/base classes so provider plug-ins can be activated via WebService/Worker. |
|
||||
| Sprint 5 | Excititor Core Foundations | src/Excititor/StellaOps.Excititor.WebService/TASKS.md | DONE (2025-10-17) | Team Excititor WebService | EXCITITOR-WEB-01-001 | Scaffold minimal API host, DI, and `/excititor/status` endpoint integrating policy, storage, export, and attestation services. |
|
||||
| Sprint 6 | Excititor Ingest & Formats | src/Excititor/StellaOps.Excititor.Worker/TASKS.md | DONE (2025-10-17) | Team Excititor Worker | EXCITITOR-WORKER-01-001 | Create Worker host with provider scheduling and logging to drive recurring pulls/reconciliation. |
|
||||
| Sprint 6 | Excititor Ingest & Formats | src/Excititor/__Libraries/StellaOps.Excititor.Formats.CSAF/TASKS.md | DONE (2025-10-17) | Team Excititor Formats | EXCITITOR-FMT-CSAF-01-001 | Implement CSAF normalizer foundation translating provider documents into `VexClaim` entries. |
|
||||
| Sprint 6 | Excititor Ingest & Formats | src/Excititor/__Libraries/StellaOps.Excititor.Formats.CycloneDX/TASKS.md | DONE (2025-10-17) | Team Excititor Formats | EXCITITOR-FMT-CYCLONE-01-001 | Implement CycloneDX VEX normalizer capturing `analysis` state and component references. |
|
||||
| Sprint 6 | Excititor Ingest & Formats | src/Excititor/__Libraries/StellaOps.Excititor.Formats.OpenVEX/TASKS.md | DONE (2025-10-17) | Team Excititor Formats | EXCITITOR-FMT-OPENVEX-01-001 | Implement OpenVEX normalizer to ingest attestations into canonical claims with provenance. |
|
||||
| Sprint 6 | Excititor Ingest & Formats | src/Excititor/__Libraries/StellaOps.Excititor.Connectors.RedHat.CSAF/TASKS.md | DONE (2025-10-17) | Team Excititor Connectors – Red Hat | EXCITITOR-CONN-RH-01-001 | Ship Red Hat CSAF provider metadata discovery enabling incremental pulls. |
|
||||
| Sprint 6 | Excititor Ingest & Formats | src/Excititor/__Libraries/StellaOps.Excititor.Connectors.RedHat.CSAF/TASKS.md | DONE (2025-10-17) | Team Excititor Connectors – Red Hat | EXCITITOR-CONN-RH-01-002 | Fetch CSAF windows with ETag handling, resume tokens, quarantine on schema errors, and persist raw docs. |
|
||||
| Sprint 6 | Excititor Ingest & Formats | src/Excititor/__Libraries/StellaOps.Excititor.Connectors.RedHat.CSAF/TASKS.md | DONE (2025-10-17) | Team Excititor Connectors – Red Hat | EXCITITOR-CONN-RH-01-003 | Populate provider trust overrides (cosign issuer, identity regex) and provenance hints for policy evaluation/logging. |
|
||||
| Sprint 6 | Excititor Ingest & Formats | src/Excititor/__Libraries/StellaOps.Excititor.Connectors.RedHat.CSAF/TASKS.md | DONE (2025-10-17) | Team Excititor Connectors – Red Hat | EXCITITOR-CONN-RH-01-004 | Persist resume cursors (last updated timestamp/document hashes) in storage and reload during fetch to avoid duplicates. |
|
||||
| Sprint 6 | Excititor Ingest & Formats | src/Excititor/__Libraries/StellaOps.Excititor.Connectors.RedHat.CSAF/TASKS.md | DONE (2025-10-17) | Team Excititor Connectors – Red Hat | EXCITITOR-CONN-RH-01-005 | Register connector in Worker/WebService DI, add scheduled jobs, and document CLI triggers for Red Hat CSAF pulls. |
|
||||
| Sprint 6 | Excititor Ingest & Formats | src/Excititor/__Libraries/StellaOps.Excititor.Connectors.RedHat.CSAF/TASKS.md | DONE (2025-10-17) | Team Excititor Connectors – Red Hat | EXCITITOR-CONN-RH-01-006 | Add CSAF normalization parity fixtures ensuring RHSA-specific metadata is preserved. |
|
||||
| Sprint 6 | Excititor Ingest & Formats | src/Excititor/__Libraries/StellaOps.Excititor.Connectors.Cisco.CSAF/TASKS.md | DONE (2025-10-17) | Team Excititor Connectors – Cisco | EXCITITOR-CONN-CISCO-01-001 | Implement Cisco CSAF endpoint discovery/auth to unlock paginated pulls. |
|
||||
| Sprint 6 | Excititor Ingest & Formats | src/Excititor/__Libraries/StellaOps.Excititor.Connectors.Cisco.CSAF/TASKS.md | DONE (2025-10-17) | Team Excititor Connectors – Cisco | EXCITITOR-CONN-CISCO-01-002 | Implement Cisco CSAF paginated fetch loop with dedupe and raw persistence support. |
|
||||
| Sprint 6 | Excititor Ingest & Formats | src/Excititor/__Libraries/StellaOps.Excititor.Connectors.SUSE.RancherVEXHub/TASKS.md | DONE (2025-10-17) | Team Excititor Connectors – SUSE | EXCITITOR-CONN-SUSE-01-001 | Build Rancher VEX Hub discovery/subscription path with offline snapshot support. |
|
||||
| Sprint 6 | Excititor Ingest & Formats | src/Excititor/__Libraries/StellaOps.Excititor.Connectors.MSRC.CSAF/TASKS.md | DONE (2025-10-17) | Team Excititor Connectors – MSRC | EXCITITOR-CONN-MS-01-001 | Deliver AAD onboarding/token cache for MSRC CSAF ingestion. |
|
||||
| Sprint 6 | Excititor Ingest & Formats | src/Excititor/__Libraries/StellaOps.Excititor.Connectors.Oracle.CSAF/TASKS.md | DONE (2025-10-17) | Team Excititor Connectors – Oracle | EXCITITOR-CONN-ORACLE-01-001 | Implement Oracle CSAF catalogue discovery with CPU calendar awareness. |
|
||||
| Sprint 6 | Excititor Ingest & Formats | src/Excititor/__Libraries/StellaOps.Excititor.Connectors.Ubuntu.CSAF/TASKS.md | DONE (2025-10-17) | Team Excititor Connectors – Ubuntu | EXCITITOR-CONN-UBUNTU-01-001 | Implement Ubuntu CSAF discovery and channel selection for USN ingestion. |
|
||||
| Sprint 6 | Excititor Ingest & Formats | src/Excititor/__Libraries/StellaOps.Excititor.Connectors.OCI.OpenVEX.Attest/TASKS.md | DONE (2025-10-18) | Team Excititor Connectors – OCI | EXCITITOR-CONN-OCI-01-001 | Wire OCI discovery/auth to fetch OpenVEX attestations for configured images. |
|
||||
| Sprint 6 | Excititor Ingest & Formats | src/Excititor/__Libraries/StellaOps.Excititor.Connectors.OCI.OpenVEX.Attest/TASKS.md | DONE (2025-10-18) | Team Excititor Connectors – OCI | EXCITITOR-CONN-OCI-01-002 | Attestation fetch & verify loop – download DSSE attestations, trigger verification, handle retries/backoff, persist raw statements. |
|
||||
| Sprint 6 | Excititor Ingest & Formats | src/Excititor/__Libraries/StellaOps.Excititor.Connectors.OCI.OpenVEX.Attest/TASKS.md | DONE (2025-10-18) | Team Excititor Connectors – OCI | EXCITITOR-CONN-OCI-01-003 | Provenance metadata & policy hooks – emit image, subject digest, issuer, and trust metadata for policy weighting/logging. |
|
||||
| Sprint 6 | Excititor Ingest & Formats | src/Cli/StellaOps.Cli/TASKS.md | DONE (2025-10-18) | DevEx/CLI | EXCITITOR-CLI-01-001 | Add `excititor` CLI verbs bridging to WebService with consistent auth and offline UX. |
|
||||
| Sprint 7 | Contextual Truth Foundations | src/Excititor/__Libraries/StellaOps.Excititor.Core/TASKS.md | DONE (2025-10-19) | Team Excititor Core & Policy | EXCITITOR-CORE-02-001 | Context signal schema prep – extend consensus models with severity/KEV/EPSS fields and update canonical serializers. |
|
||||
| Sprint 7 | Contextual Truth Foundations | src/Excititor/__Libraries/StellaOps.Excititor.Policy/TASKS.md | DONE (2025-10-19) | Team Excititor Policy | EXCITITOR-POLICY-02-001 | Scoring coefficients & weight ceilings – add α/β options, weight boosts, and validation guidance. |
|
||||
| Sprint 7 | Contextual Truth Foundations | src/Excititor/__Libraries/StellaOps.Excititor.Attestation/TASKS.md | DONE (2025-10-16) | Team Excititor Attestation | EXCITITOR-ATTEST-01-002 | Rekor v2 client integration – ship transparency log client with retries and offline queue. |
|
||||
| Sprint 9 | Scanner Core Foundations | src/Scanner/__Libraries/StellaOps.Scanner.Core/TASKS.md | DONE (2025-10-18) | Team Scanner Core | SCANNER-CORE-09-501 | Define shared DTOs (ScanJob, ProgressEvent), error taxonomy, and deterministic ID/timestamp helpers aligning with `ARCHITECTURE_SCANNER.md` §3–§4. |
|
||||
| Sprint 9 | Scanner Core Foundations | src/Scanner/__Libraries/StellaOps.Scanner.Core/TASKS.md | DONE (2025-10-18) | Team Scanner Core | SCANNER-CORE-09-502 | Observability helpers (correlation IDs, logging scopes, metric namespacing, deterministic hashes) consumed by WebService/Worker. |
|
||||
| Sprint 9 | Scanner Core Foundations | src/Scanner/__Libraries/StellaOps.Scanner.Core/TASKS.md | DONE (2025-10-18) | Team Scanner Core | SCANNER-CORE-09-503 | Security utilities: Authority client factory, OpTok caching, DPoP verifier, restart-time plug-in guardrails for scanner components. |
|
||||
| Sprint 9 | Scanner Build-time | src/Scanner/StellaOps.Scanner.Sbomer.BuildXPlugin/TASKS.md | DONE (2025-10-19) | BuildX Guild | SP9-BLDX-09-001 | Buildx driver scaffold + handshake with Scanner.Emit (local CAS). |
|
||||
| Sprint 9 | Scanner Build-time | src/Scanner/StellaOps.Scanner.Sbomer.BuildXPlugin/TASKS.md | DONE (2025-10-19) | BuildX Guild | SP9-BLDX-09-002 | OCI annotations + provenance hand-off to Attestor. |
|
||||
| Sprint 9 | Scanner Build-time | src/Scanner/StellaOps.Scanner.Sbomer.BuildXPlugin/TASKS.md | DONE (2025-10-19) | BuildX Guild | SP9-BLDX-09-003 | CI demo: minimal SBOM push & backend report wiring. |
|
||||
| Sprint 9 | Scanner Build-time | src/Scanner/StellaOps.Scanner.Sbomer.BuildXPlugin/TASKS.md | DONE (2025-10-19) | BuildX Guild | SP9-BLDX-09-004 | Stabilize descriptor nonce derivation so repeated builds emit deterministic placeholders. |
|
||||
| Sprint 9 | Scanner Build-time | src/Scanner/StellaOps.Scanner.Sbomer.BuildXPlugin/TASKS.md | DONE (2025-10-19) | BuildX Guild | SP9-BLDX-09-005 | Integrate determinism guard into GitHub/Gitea workflows and archive proof artifacts. |
|
||||
| Sprint 9 | Scanner Core Foundations | src/Scanner/StellaOps.Scanner.WebService/TASKS.md | DONE (2025-10-18) | Team Scanner WebService | SCANNER-WEB-09-101 | Minimal API host with Authority enforcement, health/ready endpoints, and restart-time plug-in loader per architecture §1, §4. |
|
||||
| Sprint 9 | Scanner Core Foundations | src/Scanner/StellaOps.Scanner.WebService/TASKS.md | DONE (2025-10-18) | Team Scanner WebService | SCANNER-WEB-09-102 | `/api/v1/scans` submission/status endpoints with deterministic IDs, validation, and cancellation support. |
|
||||
| Sprint 9 | Scanner Core Foundations | src/Scanner/StellaOps.Scanner.WebService/TASKS.md | DONE (2025-10-19) | Team Scanner WebService | SCANNER-WEB-09-104 | Configuration binding for Mongo, MinIO, queue, feature flags; startup diagnostics and fail-fast policy. |
|
||||
| Sprint 9 | Scanner Core Foundations | src/Scanner/StellaOps.Scanner.Worker/TASKS.md | DONE (2025-10-19) | Team Scanner Worker | SCANNER-WORKER-09-201 | Worker host bootstrap with Authority auth, hosted services, and graceful shutdown semantics. |
|
||||
| Sprint 9 | Scanner Core Foundations | src/Scanner/StellaOps.Scanner.Worker/TASKS.md | DONE (2025-10-19) | Team Scanner Worker | SCANNER-WORKER-09-202 | Lease/heartbeat loop with retry+jitter, poison-job quarantine, structured logging. |
|
||||
| Sprint 9 | Scanner Core Foundations | src/Scanner/StellaOps.Scanner.Worker/TASKS.md | DONE (2025-10-19) | Team Scanner Worker | SCANNER-WORKER-09-203 | Analyzer dispatch skeleton emitting deterministic stage progress and honoring cancellation tokens. |
|
||||
| Sprint 9 | Scanner Core Foundations | src/Scanner/StellaOps.Scanner.Worker/TASKS.md | DONE (2025-10-19) | Team Scanner Worker | SCANNER-WORKER-09-204 | Worker metrics (queue latency, stage duration, failure counts) with OpenTelemetry resource wiring. |
|
||||
| Sprint 9 | Scanner Core Foundations | src/Scanner/StellaOps.Scanner.Worker/TASKS.md | DONE (2025-10-19) | Team Scanner Worker | SCANNER-WORKER-09-205 | Harden heartbeat jitter so lease safety margin stays ≥3× and cover with regression tests + optional live queue smoke run. |
|
||||
| Sprint 9 | Policy Foundations | src/Policy/__Libraries/StellaOps.Policy/TASKS.md | DONE | Policy Guild | POLICY-CORE-09-001 | Policy schema + binder + diagnostics. |
|
||||
| Sprint 9 | Policy Foundations | src/Policy/__Libraries/StellaOps.Policy/TASKS.md | DONE | Policy Guild | POLICY-CORE-09-002 | Policy snapshot store + revision digests. |
|
||||
| Sprint 9 | Policy Foundations | src/Policy/__Libraries/StellaOps.Policy/TASKS.md | DONE | Policy Guild | POLICY-CORE-09-003 | `/policy/preview` API (image digest → projected verdict diff). |
|
||||
| Sprint 9 | DevOps Foundations | ops/devops/TASKS.md | DONE (2025-10-19) | DevOps Guild | DEVOPS-HELM-09-001 | Helm/Compose environment profiles (dev/staging/airgap) with deterministic digests. |
|
||||
| Sprint 9 | Docs & Governance | docs/TASKS.md | DONE (2025-10-19) | Docs Guild, DevEx | DOCS-ADR-09-001 | Establish ADR process and template. |
|
||||
| Sprint 9 | Docs & Governance | docs/TASKS.md | DONE (2025-10-19) | Docs Guild, Platform Events | DOCS-EVENTS-09-002 | Publish event schema catalog (`docs/events/`) for critical envelopes. |
|
||||
| Sprint 9 | Scanner Core Foundations | src/Scanner/__Libraries/StellaOps.Scanner.Storage/TASKS.md | DONE (2025-10-19) | Team Scanner Storage | SCANNER-STORAGE-09-301 | Mongo catalog schemas/indexes for images, layers, artifacts, jobs, lifecycle rules plus migrations. |
|
||||
| Sprint 9 | Scanner Core Foundations | src/Scanner/__Libraries/StellaOps.Scanner.Storage/TASKS.md | DONE (2025-10-19) | Team Scanner Storage | SCANNER-STORAGE-09-302 | MinIO layout, immutability policies, client abstraction, and configuration binding. |
|
||||
| Sprint 9 | Scanner Core Foundations | src/Scanner/__Libraries/StellaOps.Scanner.Storage/TASKS.md | DONE (2025-10-19) | Team Scanner Storage | SCANNER-STORAGE-09-303 | Repositories/services with dual-write feature flag, deterministic digests, TTL enforcement tests. |
|
||||
| Sprint 9 | Scanner Core Foundations | src/Scanner/__Libraries/StellaOps.Scanner.Queue/TASKS.md | DONE (2025-10-19) | Team Scanner Queue | SCANNER-QUEUE-09-401 | Queue abstraction + Redis Streams adapter with ack/claim APIs and idempotency tokens. |
|
||||
| Sprint 9 | Scanner Core Foundations | src/Scanner/__Libraries/StellaOps.Scanner.Queue/TASKS.md | DONE (2025-10-19) | Team Scanner Queue | SCANNER-QUEUE-09-402 | Pluggable backend support (Redis, NATS) with configuration binding, health probes, failover docs. |
|
||||
| Sprint 9 | Scanner Core Foundations | src/Scanner/__Libraries/StellaOps.Scanner.Queue/TASKS.md | DONE (2025-10-19) | Team Scanner Queue | SCANNER-QUEUE-09-403 | Retry + dead-letter strategy with structured logs/metrics for offline deployments. |
|
||||
| Sprint 1 | Stabilize In-Progress Foundations | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Ghsa/TASKS.md | DONE (2025-10-12) | Team Connector Normalized Versions Rollout | FEEDCONN-GHSA-02-001 | GHSA normalized versions & provenance<br>Team instructions: Read ./AGENTS.md and each module's AGENTS file. Adopt the `NormalizedVersions` array emitted by the models sprint, wiring provenance `decisionReason` where merge overrides occur. Follow ./src/FASTER_MODELING_AND_NORMALIZATION.md; report via src/Concelier/__Libraries/StellaOps.Concelier.Merge/TASKS.md (FEEDMERGE-COORD-02-900). Progress 2025-10-11: GHSA/OSV emit normalized arrays with refreshed fixtures; CVE mapper now surfaces SemVer normalized ranges; NVD/KEV adoption pending; outstanding follow-ups include FEEDSTORAGE-DATA-02-001, FEEDMERGE-ENGINE-02-002, and rolling `tools/FixtureUpdater` updates across connectors.<br>Progress 2025-10-20: Coordination matrix + rollout dashboard refreshed; upcoming deadlines tracked (Cccs/Cisco 2025-10-21, CertBund 2025-10-22, ICS-CISA 2025-10-23, KISA 2025-10-24) with escalation path documented in FEEDMERGE-COORD-02-900.|
|
||||
| Sprint 1 | Stabilize In-Progress Foundations | src/Concelier/StellaOps.Concelier.WebService/TASKS.md | DONE (2025-10-19) | Team WebService & Authority | FEEDWEB-OPS-01-006 | Rename plugin drop directory to namespaced path<br>Build outputs now point at `StellaOps.Concelier.PluginBinaries`/`StellaOps.Authority.PluginBinaries`; defaults/docs/tests updated to reflect the new layout. |
|
||||
| Sprint 7 | Contextual Truth Foundations | src/Excititor/__Libraries/StellaOps.Excititor.Storage.Mongo/TASKS.md | DONE (2025-10-19) | Team Excititor Storage | EXCITITOR-STORAGE-02-001 | Statement events & scoring signals – immutable VEX statements store, consensus signal fields, and migration `20251019-consensus-signals-statements` with tests (`dotnet test src/Excititor/__Tests/StellaOps.Excititor.Core.Tests/StellaOps.Excititor.Core.Tests.csproj`, `dotnet test src/Excititor/__Tests/StellaOps.Excititor.Storage.Mongo.Tests/StellaOps.Excititor.Storage.Mongo.Tests.csproj`). |
|
||||
| Sprint 7 | Contextual Truth Foundations | src/Concelier/__Libraries/StellaOps.Concelier.Core/TASKS.md | DONE (2025-10-19) | Team Core Engine & Storage Analytics | FEEDCORE-ENGINE-07-001 | Advisory event log & asOf queries – surface immutable statements and replay capability. |
|
||||
| Sprint 7 | Contextual Truth Foundations | src/Concelier/StellaOps.Concelier.WebService/TASKS.md | DONE (2025-10-19) | Concelier WebService Guild | FEEDWEB-EVENTS-07-001 | Advisory event replay API – expose `/concelier/advisories/{key}/replay` with `asOf` filter, hex hashes, and conflict data. |
|
||||
| Sprint 7 | Contextual Truth Foundations | src/Concelier/__Libraries/StellaOps.Concelier.Merge/TASKS.md | DONE (2025-10-20) | BE-Merge | FEEDMERGE-ENGINE-07-001 | Conflict sets & explainers – persist conflict materialization and replay hashes for merge decisions. |
|
||||
| Sprint 8 | Mongo strengthening | src/Concelier/__Libraries/StellaOps.Concelier.Storage.Mongo/TASKS.md | DONE (2025-10-19) | Team Normalization & Storage Backbone | FEEDSTORAGE-MONGO-08-001 | Causal-consistent Concelier storage sessions<br>Scoped session facilitator registered, repositories accept optional session handles, and replica-set failover tests verify read-your-write + monotonic reads. |
|
||||
| Sprint 8 | Mongo strengthening | src/Authority/StellaOps.Authority/TASKS.md | DONE (2025-10-19) | Authority Core & Storage Guild | AUTHSTORAGE-MONGO-08-001 | Harden Authority Mongo usage<br>Scoped Mongo sessions with majority read/write concerns wired through stores and GraphQL/HTTP pipelines; replica-set election regression validated. |
|
||||
| Sprint 8 | Mongo strengthening | src/Excititor/__Libraries/StellaOps.Excititor.Storage.Mongo/TASKS.md | DONE (2025-10-19) | Team Excititor Storage | EXCITITOR-STORAGE-MONGO-08-001 | Causal consistency for Excititor repositories<br>Session-scoped repositories shipped with new Mongo records, orchestrators/workers now share scoped sessions, and replica-set failover coverage added via `dotnet test src/Excititor/__Tests/StellaOps.Excititor.Storage.Mongo.Tests/StellaOps.Excititor.Storage.Mongo.Tests.csproj`. |
|
||||
| Sprint 8 | Platform Maintenance | src/Excititor/__Libraries/StellaOps.Excititor.Storage.Mongo/TASKS.md | DONE (2025-10-19) | Team Excititor Storage | EXCITITOR-STORAGE-03-001 | Statement backfill tooling – shipped admin backfill endpoint, CLI hook (`stellaops excititor backfill-statements`), integration tests, and operator runbook (`docs/dev/EXCITITOR_STATEMENT_BACKFILL.md`). |
|
||||
| Sprint 8 | Mirror Distribution | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Exporter.Json/TASKS.md | DONE (2025-10-19) | Concelier Export Guild | CONCELIER-EXPORT-08-201 | Mirror bundle + domain manifest – produce signed JSON aggregates for `*.stella-ops.org` mirrors. |
|
||||
| Sprint 8 | Mirror Distribution | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Exporter.TrivyDb/TASKS.md | DONE (2025-10-19) | Concelier Export Guild | CONCELIER-EXPORT-08-202 | Mirror-ready Trivy DB bundles – mirror options emit per-domain manifests/metadata/db archives with deterministic digests for downstream sync. |
|
||||
| Sprint 8 | Mirror Distribution | src/Concelier/StellaOps.Concelier.WebService/TASKS.md | DONE (2025-10-20) | Concelier WebService Guild | CONCELIER-WEB-08-201 | Mirror distribution endpoints – expose domain-scoped index/download APIs with auth/quota. |
|
||||
| Sprint 8 | Mirror Distribution | ops/devops/TASKS.md | DONE (2025-10-19) | DevOps Guild | DEVOPS-MIRROR-08-001 | Managed mirror deployments for `*.stella-ops.org` – Helm/Compose overlays, CDN, runbooks. |
|
||||
| Sprint 8 | Plugin Infrastructure | src/__Libraries/StellaOps.Plugin/TASKS.md | DONE (2025-10-20) | Plugin Platform Guild, Authority Core | PLUGIN-DI-08-003 | Refactor Authority identity-provider registry to resolve scoped plugin services on-demand.<br>Introduce factory pattern aligned with scoped lifetimes decided in coordination workshop. |
|
||||
| Sprint 8 | Plugin Infrastructure | src/__Libraries/StellaOps.Plugin/TASKS.md | DONE (2025-10-20) | Plugin Platform Guild, Authority Core | PLUGIN-DI-08-004 | Update Authority plugin loader to activate registrars with DI support and scoped service awareness.<br>Add two-phase initialization allowing scoped dependencies post-container build. |
|
||||
| Sprint 8 | Plugin Infrastructure | src/__Libraries/StellaOps.Plugin/TASKS.md | DONE (2025-10-20) | Plugin Platform Guild, Authority Core | PLUGIN-DI-08-005 | Provide scoped-safe bootstrap execution for Authority plugins.<br>Implement scope-per-run pattern for hosted bootstrap tasks and document migration guidance. |
|
||||
| Sprint 10 | DevOps Security | ops/devops/TASKS.md | DONE (2025-10-20) | DevOps Guild | DEVOPS-SEC-10-301 | Address NU1902/NU1903 advisories for `MongoDB.Driver` 2.12.0 and `SharpCompress` 0.23.0; Wave 0A prerequisites confirmed complete before remediation work. |
|
||||
| Sprint 11 | Signing Chain Bring-up | src/Authority/StellaOps.Authority/TASKS.md | DONE (2025-10-20) | Authority Core & Security Guild | AUTH-DPOP-11-001 | Implement DPoP proof validation + nonce handling for high-value audiences per architecture. |
|
||||
| Sprint 15 | Notify Foundations | src/Notify/StellaOps.Notify.WebService/TASKS.md | DONE (2025-10-19) | Notify WebService Guild | NOTIFY-WEB-15-103 | Delivery history & test-send endpoints. |
|
||||
| Sprint 15 | Notify Foundations | src/Notify/__Libraries/StellaOps.Notify.Connectors.Slack/TASKS.md | DONE (2025-10-20) | Notify Connectors Guild | NOTIFY-CONN-SLACK-15-502 | Slack health/test-send support. |
|
||||
| Sprint 15 | Notify Foundations | src/Notify/__Libraries/StellaOps.Notify.Connectors.Teams/TASKS.md | DONE (2025-10-20) | Notify Connectors Guild | NOTIFY-CONN-TEAMS-15-602 | Teams health/test-send support. |
|
||||
| Sprint 15 | Notify Foundations | src/Notify/__Libraries/StellaOps.Notify.Connectors.Teams/TASKS.md | DONE (2025-10-20) | Notify Connectors Guild | NOTIFY-CONN-TEAMS-15-604 | Teams health endpoint metadata alignment. |
|
||||
| Sprint 15 | Notify Foundations | src/Notify/__Libraries/StellaOps.Notify.Connectors.Slack/TASKS.md | DONE (2025-10-20) | Notify Connectors Guild | NOTIFY-CONN-SLACK-15-503 | Package Slack connector as restart-time plug-in (manifest + host registration). |
|
||||
| Sprint 15 | Notify Foundations | src/Notify/__Libraries/StellaOps.Notify.Connectors.Teams/TASKS.md | DONE (2025-10-20) | Notify Connectors Guild | NOTIFY-CONN-TEAMS-15-603 | Package Teams connector as restart-time plug-in (manifest + host registration). |
|
||||
| Sprint 15 | Notify Foundations | src/Notify/__Libraries/StellaOps.Notify.Connectors.Email/TASKS.md | DONE (2025-10-20) | Notify Connectors Guild | NOTIFY-CONN-EMAIL-15-703 | Package Email connector as restart-time plug-in (manifest + host registration). |
|
||||
| Sprint 15 | Notify Foundations | src/Scanner/StellaOps.Scanner.WebService/TASKS.md | DONE (2025-10-20) | Scanner WebService Guild | SCANNER-EVENTS-15-201 | Emit `scanner.report.ready` + `scanner.scan.completed` events. |
|
||||
| Sprint 15 | Notify Foundations | src/Notify/__Libraries/StellaOps.Notify.Connectors.Webhook/TASKS.md | DONE (2025-10-20) | Notify Connectors Guild | NOTIFY-CONN-WEBHOOK-15-803 | Package Webhook connector as restart-time plug-in (manifest + host registration). |
|
||||
| Sprint 16 | Scheduler Intelligence | src/Scheduler/__Libraries/StellaOps.Scheduler.Models/TASKS.md | DONE (2025-10-20) | Scheduler Models Guild | SCHED-MODELS-16-103 | Versioning/migration helpers for schedules/runs. |
|
||||
| Sprint 16 | Scheduler Intelligence | src/Scheduler/__Libraries/StellaOps.Scheduler.Queue/TASKS.md | DONE (2025-10-20) | Scheduler Queue Guild | SCHED-QUEUE-16-401 | Queue abstraction + Redis Streams adapter. |
|
||||
| Sprint 16 | Scheduler Intelligence | src/Scheduler/__Libraries/StellaOps.Scheduler.Queue/TASKS.md | DONE (2025-10-20) | Scheduler Queue Guild | SCHED-QUEUE-16-402 | NATS JetStream adapter with health probes. |
|
||||
| Sprint 16 | Scheduler Intelligence | src/Scheduler/__Libraries/StellaOps.Scheduler.ImpactIndex/TASKS.md | DONE (2025-10-20) | Scheduler ImpactIndex Guild | SCHED-IMPACT-16-300 | **STUB** ImpactIndex ingest/query using fixtures (to be removed by SP16 completion). |
|
||||
88
docs/implplan/SPRINTS_PRIOR_20251021.md
Normal file
88
docs/implplan/SPRINTS_PRIOR_20251021.md
Normal file
@@ -0,0 +1,88 @@
|
||||
This file describe implementation of Stella Ops (docs/README.md). Implementation must respect rules from AGENTS.md (read if you have not).
|
||||
|
||||
| Sprint | Theme | Tasks File Path | Status | Type of Specialist | Task ID | Task Description |
|
||||
| --- | --- | --- | --- | --- | --- | --- |
|
||||
| Sprint 7 | Contextual Truth Foundations | docs/TASKS.md | DONE (2025-10-22) | Docs Guild, Concelier WebService | DOCS-CONCELIER-07-201 | Final editorial review and publish pass for Concelier authority toggle documentation (Quickstart + operator guide). |
|
||||
| Sprint 7 | Contextual Truth Foundations | src/Excititor/StellaOps.Excititor.WebService/TASKS.md | DONE (2025-10-20) | Team Excititor WebService | EXCITITOR-WEB-01-002 | Ingest & reconcile endpoints – scope-enforced `/excititor/init`, `/excititor/ingest/run`, `/excititor/ingest/resume`, `/excititor/reconcile`; regression via `dotnet test … --filter FullyQualifiedName~IngestEndpointsTests`. |
|
||||
| Sprint 7 | Contextual Truth Foundations | src/Excititor/StellaOps.Excititor.WebService/TASKS.md | DONE (2025-10-20) | Team Excititor WebService | EXCITITOR-WEB-01-004 | Resolve API & signed responses – expose `/excititor/resolve`, return signed consensus/score envelopes, document auth. |
|
||||
| Sprint 7 | Contextual Truth Foundations | src/Excititor/StellaOps.Excititor.Worker/TASKS.md | DONE (2025-10-21) | Team Excititor Worker | EXCITITOR-WORKER-01-004 | TTL refresh & stability damper – schedule re-resolve loops and guard against status flapping. |
|
||||
| Sprint 7 | Contextual Truth Foundations | src/Concelier/__Libraries/StellaOps.Concelier.Core/TASKS.md | DONE (2025-10-21) | Team Core Engine & Data Science | FEEDCORE-ENGINE-07-002 | Noise prior computation service – learn false-positive priors and expose deterministic summaries. |
|
||||
| Sprint 7 | Contextual Truth Foundations | src/Concelier/__Libraries/StellaOps.Concelier.Core/TASKS.md | DONE (2025-10-21) | Team Core Engine & Storage Analytics | FEEDCORE-ENGINE-07-003 | Unknown state ledger & confidence seeding – persist unknown flags, seed confidence bands, expose query surface. |
|
||||
| Sprint 7 | Contextual Truth Foundations | src/Excititor/StellaOps.Excititor.WebService/TASKS.md | DONE (2025-10-19) | Team Excititor WebService | EXCITITOR-WEB-01-005 | Mirror distribution endpoints – expose download APIs for downstream Excititor instances. |
|
||||
| Sprint 7 | Contextual Truth Foundations | src/Excititor/__Libraries/StellaOps.Excititor.Export/TASKS.md | DONE (2025-10-21) | Team Excititor Export | EXCITITOR-EXPORT-01-005 | Score & resolve envelope surfaces – include signed consensus/score artifacts in exports. |
|
||||
| Sprint 7 | Contextual Truth Foundations | src/Excititor/__Libraries/StellaOps.Excititor.Export/TASKS.md | DONE (2025-10-21) | Team Excititor Export | EXCITITOR-EXPORT-01-006 | Quiet provenance packaging – attach quieted-by statement IDs, signers, justification codes to exports and attestations. |
|
||||
| Sprint 7 | Contextual Truth Foundations | src/Excititor/__Libraries/StellaOps.Excititor.Export/TASKS.md | DONE (2025-10-21) | Team Excititor Export | EXCITITOR-EXPORT-01-007 | Mirror bundle + domain manifest – publish signed consensus bundles for mirrors. |
|
||||
| Sprint 7 | Contextual Truth Foundations | src/Excititor/StellaOps.Excititor.Connectors.StellaOpsMirror/TASKS.md | DONE (2025-10-21) | Excititor Connectors – Stella | EXCITITOR-CONN-STELLA-07-001 | Excititor mirror connector – ingest signed mirror bundles and map to VexClaims with resume handling. |
|
||||
| Sprint 7 | Contextual Truth Foundations | src/Concelier/__Libraries/StellaOps.Concelier.Storage.Mongo/TASKS.md | DONE (2025-10-19) | Team Normalization & Storage Backbone | FEEDSTORAGE-DATA-07-001 | Advisory statement & conflict collections – provision Mongo schema/indexes for event-sourced merge. |
|
||||
| Sprint 7 | Contextual Truth Foundations | src/Web/StellaOps.Web/TASKS.md | DONE (2025-10-21) | UX Specialist, Angular Eng | WEB1.TRIVY-SETTINGS-TESTS | Add headless UI test run (`ng test --watch=false`) and document prerequisites once Angular tooling is chained up. |
|
||||
| Sprint 8 | Mirror Distribution | src/Concelier/__Libraries/StellaOps.Concelier.Connector.StellaOpsMirror/TASKS.md | DONE (2025-10-20) | BE-Conn-Stella | FEEDCONN-STELLA-08-001 | Concelier mirror connector – fetch mirror manifest, verify signatures, and hydrate canonical DTOs with resume support. |
|
||||
| Sprint 8 | Mirror Distribution | src/Concelier/__Libraries/StellaOps.Concelier.Connector.StellaOpsMirror/TASKS.md | DONE (2025-10-20) | BE-Conn-Stella | FEEDCONN-STELLA-08-002 | Map mirror payloads into canonical advisory DTOs with provenance referencing mirror domain + original source metadata. |
|
||||
| Sprint 8 | Mirror Distribution | src/Concelier/__Libraries/StellaOps.Concelier.Connector.StellaOpsMirror/TASKS.md | DONE (2025-10-20) | BE-Conn-Stella | FEEDCONN-STELLA-08-003 | Add incremental cursor + resume support (per-export fingerprint) and document configuration for downstream Concelier instances. |
|
||||
| Sprint 8 | Plugin Infrastructure | src/__Libraries/StellaOps.Plugin/TASKS.md | DONE (2025-10-21) | Plugin Platform Guild | PLUGIN-DI-08-001 | Scoped service support in plugin bootstrap – added dynamic plugin tests ensuring `[ServiceBinding]` metadata flows through plugin hosts and remains idempotent. |
|
||||
| Sprint 8 | Plugin Infrastructure | src/__Libraries/StellaOps.Plugin/TASKS.md | DONE (2025-10-20) | Plugin Platform Guild, Authority Core | PLUGIN-DI-08-002.COORD | Authority scoped-service integration handshake<br>Workshop concluded 2025-10-20 15:00–16:05 UTC; decisions + follow-ups recorded in `docs/dev/authority-plugin-di-coordination.md`. |
|
||||
| Sprint 8 | Plugin Infrastructure | src/__Libraries/StellaOps.Plugin/TASKS.md | DONE (2025-10-20) | Plugin Platform Guild, Authority Core | PLUGIN-DI-08-002 | Authority plugin integration updates – scoped identity-provider services with registry handles; regression coverage via scoped registrar/unit tests. |
|
||||
| Sprint 8 | Plugin Infrastructure | src/Authority/StellaOps.Authority/TASKS.md | DONE (2025-10-20) | Authority Core, Plugin Platform Guild | AUTH-PLUGIN-COORD-08-002 | Coordinate scoped-service adoption for Authority plug-in registrars<br>Workshop notes and follow-up backlog captured 2025-10-20 in `docs/dev/authority-plugin-di-coordination.md`. |
|
||||
| Sprint 9 | Scanner Core Foundations | src/Scanner/StellaOps.Scanner.WebService/TASKS.md | DONE (2025-10-19) | Team Scanner WebService | SCANNER-WEB-09-103 | Progress streaming (SSE/JSONL) with correlation IDs and ISO-8601 UTC timestamps, documented in API reference. |
|
||||
| Sprint 9 | Scanner Core Foundations | src/Scanner/StellaOps.Scanner.WebService/TASKS.md | DONE (2025-10-19) | Team Scanner WebService | SCANNER-POLICY-09-105 | Policy snapshot loader + schema + OpenAPI (YAML ignore rules, VEX include/exclude, vendor precedence). |
|
||||
| Sprint 9 | Scanner Core Foundations | src/Scanner/StellaOps.Scanner.WebService/TASKS.md | DONE (2025-10-19) | Team Scanner WebService | SCANNER-POLICY-09-106 | `/reports` verdict assembly (Feedser+Vexer+Policy) + signed response envelope. |
|
||||
| Sprint 9 | Scanner Core Foundations | src/Scanner/StellaOps.Scanner.WebService/TASKS.md | DONE (2025-10-19) | Team Scanner WebService | SCANNER-POLICY-09-107 | Expose score inputs, config version, and quiet provenance in `/reports` JSON and signed payload. |
|
||||
| Sprint 9 | DevOps Foundations | ops/devops/TASKS.md | DONE (2025-10-21) | DevOps Guild, Scanner WebService Guild | DEVOPS-SCANNER-09-204 | Surface `SCANNER__EVENTS__*` env config across Compose/Helm and document overrides. |
|
||||
| Sprint 9 | DevOps Foundations | ops/devops/TASKS.md | DONE (2025-10-21) | DevOps Guild, Notify Guild | DEVOPS-SCANNER-09-205 | Notify smoke job validates Redis stream + Notify deliveries after staging deploys. |
|
||||
| Sprint 9 | Policy Foundations | src/Policy/__Libraries/StellaOps.Policy/TASKS.md | DONE (2025-10-19) | Policy Guild | POLICY-CORE-09-004 | Versioned scoring config with schema validation, trust table, and golden fixtures. |
|
||||
| Sprint 9 | Policy Foundations | src/Policy/__Libraries/StellaOps.Policy/TASKS.md | DONE (2025-10-19) | Policy Guild | POLICY-CORE-09-005 | Scoring/quiet engine – compute score, enforce VEX-only quiet rules, emit inputs and provenance. |
|
||||
| Sprint 9 | Policy Foundations | src/Policy/__Libraries/StellaOps.Policy/TASKS.md | DONE (2025-10-19) | Policy Guild | POLICY-CORE-09-006 | Unknown state & confidence decay – deterministic bands surfaced in policy outputs. |
|
||||
| Sprint 9 | Docs & Governance | docs/TASKS.md | DONE (2025-10-21) | Platform Events Guild | PLATFORM-EVENTS-09-401 | Embed canonical event samples into contract/integration tests and ensure CI validates payloads against published schemas. |
|
||||
| Sprint 10 | Benchmarks | src/Bench/StellaOps.Bench/TASKS.md | DONE (2025-10-21) | Bench Guild, Language Analyzer Guild | BENCH-SCANNER-10-002 | Wire real language analyzers into bench harness & refresh baselines post-implementation. |
|
||||
| Sprint 10 | Scanner Analyzers & SBOM | src/Scanner/__Libraries/StellaOps.Scanner.Analyzers.Lang/TASKS.md | DONE (2025-10-21) | Language Analyzer Guild | SCANNER-ANALYZERS-LANG-10-302 | Node analyzer handling workspaces/symlinks emitting `pkg:npm`. |
|
||||
| Sprint 10 | Scanner Analyzers & SBOM | src/Scanner/__Libraries/StellaOps.Scanner.Analyzers.Lang/TASKS.md | DONE (2025-10-21) | Language Analyzer Guild | SCANNER-ANALYZERS-LANG-10-303 | Python analyzer reading `*.dist-info`, RECORD hashes, entry points. |
|
||||
| Sprint 10 | Scanner Analyzers & SBOM | src/Scanner/__Libraries/StellaOps.Scanner.Analyzers.Lang/TASKS.md | DONE (2025-10-22) | Language Analyzer Guild | SCANNER-ANALYZERS-LANG-10-304 | Go analyzer leveraging buildinfo for `pkg:golang` components. |
|
||||
| Sprint 10 | Scanner Analyzers & SBOM | src/Scanner/__Libraries/StellaOps.Scanner.Analyzers.Lang.Go/TASKS.md | DONE (2025-10-22) | Language Analyzer Guild | SCANNER-ANALYZERS-LANG-10-304E | Plumb Go heuristic counter into Scanner metrics pipeline and alerting. |
|
||||
| Sprint 10 | Scanner Analyzers & SBOM | src/Scanner/__Libraries/StellaOps.Scanner.Analyzers.Lang/TASKS.md | DONE (2025-10-22) | Language Analyzer Guild | SCANNER-ANALYZERS-LANG-10-305 | .NET analyzer parsing `*.deps.json`, assembly metadata, RID variants. |
|
||||
| Sprint 10 | Scanner Analyzers & SBOM | src/Scanner/__Libraries/StellaOps.Scanner.Analyzers.Lang/TASKS.md | DONE (2025-10-22) | Language Analyzer Guild | SCANNER-ANALYZERS-LANG-10-306 | Rust analyzer detecting crates or falling back to `bin:{sha256}`. |
|
||||
| Sprint 10 | Scanner Analyzers & SBOM | src/Scanner/__Libraries/StellaOps.Scanner.Analyzers.Lang/TASKS.md | DONE (2025-10-19) | Language Analyzer Guild | SCANNER-ANALYZERS-LANG-10-307 | Shared language evidence helpers + usage flag propagation. |
|
||||
| Sprint 10 | Scanner Analyzers & SBOM | src/Scanner/__Libraries/StellaOps.Scanner.Analyzers.Lang/TASKS.md | DONE (2025-10-19) | Language Analyzer Guild | SCANNER-ANALYZERS-LANG-10-308 | Determinism + fixture harness for language analyzers. |
|
||||
| Sprint 10 | Scanner Analyzers & SBOM | src/Scanner/__Libraries/StellaOps.Scanner.Analyzers.Lang/TASKS.md | DONE (2025-10-21) | Language Analyzer Guild | SCANNER-ANALYZERS-LANG-10-309 | Package language analyzers as restart-time plug-ins (manifest + host registration). |
|
||||
| Sprint 10 | Scanner Analyzers & SBOM | src/Scanner/__Libraries/StellaOps.Scanner.Emit/TASKS.md | DONE (2025-10-22) | Emit Guild | SCANNER-EMIT-10-601 | Compose inventory SBOM (CycloneDX JSON/Protobuf) from layer fragments. |
|
||||
| Sprint 10 | Scanner Analyzers & SBOM | src/Scanner/__Libraries/StellaOps.Scanner.Emit/TASKS.md | DONE (2025-10-22) | Emit Guild | SCANNER-EMIT-10-602 | Compose usage SBOM leveraging EntryTrace to flag actual usage. |
|
||||
| Sprint 10 | Scanner Analyzers & SBOM | src/Scanner/__Libraries/StellaOps.Scanner.Emit/TASKS.md | DONE (2025-10-22) | Emit Guild | SCANNER-EMIT-10-603 | Generate BOM index sidecar (purl table + roaring bitmap + usage flag). |
|
||||
| Sprint 10 | Scanner Analyzers & SBOM | src/Scanner/__Libraries/StellaOps.Scanner.Emit/TASKS.md | DONE (2025-10-22) | Emit Guild | SCANNER-EMIT-10-604 | Package artifacts for export + attestation with deterministic manifests. |
|
||||
| Sprint 10 | Scanner Analyzers & SBOM | src/Scanner/__Libraries/StellaOps.Scanner.Emit/TASKS.md | DONE (2025-10-22) | Emit Guild | SCANNER-EMIT-10-605 | Emit BOM-Index sidecar schema/fixtures (CRITICAL PATH for SP16). |
|
||||
| Sprint 10 | Scanner Analyzers & SBOM | src/Scanner/__Libraries/StellaOps.Scanner.Emit/TASKS.md | DONE (2025-10-22) | Emit Guild | SCANNER-EMIT-10-606 | Usage view bit flags integrated with EntryTrace. |
|
||||
| Sprint 10 | Scanner Analyzers & SBOM | src/Scanner/__Libraries/StellaOps.Scanner.Emit/TASKS.md | DONE (2025-10-22) | Emit Guild | SCANNER-EMIT-10-607 | Embed scoring inputs, confidence band, and quiet provenance in CycloneDX/DSSE artifacts. |
|
||||
| Sprint 10 | Scanner Analyzers & SBOM | src/Scanner/__Libraries/StellaOps.Scanner.Cache/TASKS.md | DONE (2025-10-19) | Scanner Cache Guild | SCANNER-CACHE-10-101 | Implement layer cache store keyed by layer digest with metadata retention per architecture §3.3. |
|
||||
| Sprint 10 | Scanner Analyzers & SBOM | src/Scanner/__Libraries/StellaOps.Scanner.Cache/TASKS.md | DONE (2025-10-19) | Scanner Cache Guild | SCANNER-CACHE-10-102 | Build file CAS with dedupe, TTL enforcement, and offline import/export hooks. |
|
||||
| Sprint 10 | Scanner Analyzers & SBOM | src/Scanner/__Libraries/StellaOps.Scanner.Cache/TASKS.md | DONE (2025-10-19) | Scanner Cache Guild | SCANNER-CACHE-10-103 | Expose cache metrics/logging and configuration toggles for warm/cold thresholds. |
|
||||
| Sprint 10 | Scanner Analyzers & SBOM | src/Scanner/__Libraries/StellaOps.Scanner.Cache/TASKS.md | DONE (2025-10-19) | Scanner Cache Guild | SCANNER-CACHE-10-104 | Implement cache invalidation workflows (layer delete, TTL expiry, diff invalidation). |
|
||||
| Sprint 10 | Scanner Analyzers & SBOM | src/Scanner/__Libraries/StellaOps.Scanner.Analyzers.OS/TASKS.md | DONE (2025-10-19) | OS Analyzer Guild | SCANNER-ANALYZERS-OS-10-201 | Alpine/apk analyzer emitting deterministic components with provenance. |
|
||||
| Sprint 10 | Scanner Analyzers & SBOM | src/Scanner/__Libraries/StellaOps.Scanner.Analyzers.OS/TASKS.md | DONE (2025-10-19) | OS Analyzer Guild | SCANNER-ANALYZERS-OS-10-202 | Debian/dpkg analyzer mapping packages to purl identity with evidence. |
|
||||
| Sprint 10 | Scanner Analyzers & SBOM | src/Scanner/__Libraries/StellaOps.Scanner.Analyzers.OS/TASKS.md | DONE (2025-10-19) | OS Analyzer Guild | SCANNER-ANALYZERS-OS-10-203 | RPM analyzer capturing EVR, file listings, provenance. |
|
||||
| Sprint 10 | Scanner Analyzers & SBOM | src/Scanner/__Libraries/StellaOps.Scanner.Analyzers.OS/TASKS.md | DONE (2025-10-19) | OS Analyzer Guild | SCANNER-ANALYZERS-OS-10-204 | Shared OS evidence helpers for package identity + provenance. |
|
||||
| Sprint 10 | Scanner Analyzers & SBOM | src/Scanner/__Libraries/StellaOps.Scanner.Analyzers.OS/TASKS.md | DONE (2025-10-19) | OS Analyzer Guild | SCANNER-ANALYZERS-OS-10-205 | Vendor metadata enrichment (source packages, license, CVE hints). |
|
||||
| Sprint 10 | Scanner Analyzers & SBOM | src/Scanner/__Libraries/StellaOps.Scanner.Analyzers.OS/TASKS.md | DONE (2025-10-19) | OS Analyzer Guild | SCANNER-ANALYZERS-OS-10-206 | Determinism harness + fixtures for OS analyzers. |
|
||||
| Sprint 10 | Scanner Analyzers & SBOM | src/Scanner/__Libraries/StellaOps.Scanner.Analyzers.OS/TASKS.md | DONE (2025-10-19) | OS Analyzer Guild | SCANNER-ANALYZERS-OS-10-207 | Package OS analyzers as restart-time plug-ins (manifest + host registration). |
|
||||
| Sprint 10 | Scanner Analyzers & SBOM | src/Scanner/__Libraries/StellaOps.Scanner.Analyzers.Lang/TASKS.md | DONE (2025-10-19) | Language Analyzer Guild | SCANNER-ANALYZERS-LANG-10-301 | Java analyzer emitting `pkg:maven` with provenance. |
|
||||
| Sprint 10 | Scanner Analyzers & SBOM | src/Scanner/__Libraries/StellaOps.Scanner.EntryTrace/TASKS.md | DONE (2025-10-19) | EntryTrace Guild | SCANNER-ENTRYTRACE-10-401 | POSIX shell AST parser with deterministic output. |
|
||||
| Sprint 10 | Scanner Analyzers & SBOM | src/Scanner/__Libraries/StellaOps.Scanner.EntryTrace/TASKS.md | DONE (2025-10-19) | EntryTrace Guild | SCANNER-ENTRYTRACE-10-402 | Command resolution across layered rootfs with evidence attribution. |
|
||||
| Sprint 10 | Scanner Analyzers & SBOM | src/Scanner/__Libraries/StellaOps.Scanner.EntryTrace/TASKS.md | DONE (2025-10-19) | EntryTrace Guild | SCANNER-ENTRYTRACE-10-403 | Interpreter tracing for shell wrappers to Python/Node/Java launchers. |
|
||||
| Sprint 10 | Scanner Analyzers & SBOM | src/Scanner/__Libraries/StellaOps.Scanner.EntryTrace/TASKS.md | DONE (2025-10-19) | EntryTrace Guild | SCANNER-ENTRYTRACE-10-404 | Python entry analyzer (venv shebang, module invocation, usage flag). |
|
||||
| Sprint 10 | Scanner Analyzers & SBOM | src/Scanner/__Libraries/StellaOps.Scanner.EntryTrace/TASKS.md | DONE (2025-10-19) | EntryTrace Guild | SCANNER-ENTRYTRACE-10-405 | Node/Java launcher analyzer capturing script/jar targets. |
|
||||
| Sprint 10 | Scanner Analyzers & SBOM | src/Scanner/__Libraries/StellaOps.Scanner.EntryTrace/TASKS.md | DONE (2025-10-19) | EntryTrace Guild | SCANNER-ENTRYTRACE-10-406 | Explainability + diagnostics for unresolved constructs with metrics. |
|
||||
| Sprint 10 | Scanner Analyzers & SBOM | src/Scanner/__Libraries/StellaOps.Scanner.EntryTrace/TASKS.md | DONE (2025-10-19) | EntryTrace Guild | SCANNER-ENTRYTRACE-10-407 | Package EntryTrace analyzers as restart-time plug-ins (manifest + host registration). |
|
||||
| Sprint 10 | Scanner Analyzers & SBOM | src/Scanner/__Libraries/StellaOps.Scanner.Diff/TASKS.md | DONE (2025-10-19) | Diff Guild | SCANNER-DIFF-10-501 | Build component differ tracking add/remove/version changes with deterministic ordering. |
|
||||
| Sprint 10 | Scanner Analyzers & SBOM | src/Scanner/__Libraries/StellaOps.Scanner.Diff/TASKS.md | DONE (2025-10-19) | Diff Guild | SCANNER-DIFF-10-502 | Attribute diffs to introducing/removing layers including provenance evidence. |
|
||||
| Sprint 10 | Scanner Analyzers & SBOM | src/Scanner/__Libraries/StellaOps.Scanner.Diff/TASKS.md | DONE (2025-10-19) | Diff Guild | SCANNER-DIFF-10-503 | Produce JSON diff output for inventory vs usage views aligned with API contract. |
|
||||
| Sprint 10 | Samples | samples/TASKS.md | DONE (2025-10-20) | Samples Guild, Scanner Team | SAMPLES-10-001 | Sample images with SBOM/BOM-Index sidecars. |
|
||||
| Sprint 10 | DevOps Perf | ops/devops/TASKS.md | DONE (2025-10-22) | DevOps Guild | DEVOPS-PERF-10-001 | Perf smoke job ensuring <5 s SBOM compose. |
|
||||
| Sprint 10 | DevOps Perf | ops/devops/TASKS.md | DONE (2025-10-23) | DevOps Guild | DEVOPS-PERF-10-002 | Publish analyzer bench metrics to Grafana/perf workbook and alarm on ≥20 % regressions. |
|
||||
| Sprint 10 | Policy Samples | samples/TASKS.md | DONE (2025-10-23) | Samples Guild, Policy Guild | SAMPLES-13-004 | Add policy preview/report fixtures showing confidence bands and unknown-age tags. |
|
||||
| Sprint 10 | Policy Samples | src/Web/StellaOps.Web/TASKS.md | DONE (2025-10-23) | UI Guild | WEB-POLICY-FIXTURES-10-001 | Wire policy preview/report doc fixtures into UI harness (test utility or Storybook substitute) with type bindings and validation guard so UI stays aligned with documented payloads. |
|
||||
| Sprint 11 | Signing Chain Bring-up | src/Signer/StellaOps.Signer/TASKS.md | DONE (2025-10-21) | Signer Guild | SIGNER-API-11-101 | `/sign/dsse` pipeline with Authority auth, PoE introspection, release verification, DSSE signing. |
|
||||
| Sprint 11 | Signing Chain Bring-up | src/Signer/StellaOps.Signer/TASKS.md | DONE (2025-10-21) | Signer Guild | SIGNER-REF-11-102 | `/verify/referrers` endpoint with OCI lookup, caching, and policy enforcement. |
|
||||
| Sprint 11 | Signing Chain Bring-up | src/Signer/StellaOps.Signer/TASKS.md | DONE (2025-10-21) | Signer Guild | SIGNER-QUOTA-11-103 | Enforce plan quotas, concurrency/QPS limits, artifact size caps with metrics/audit logs. |
|
||||
| Sprint 11 | Signing Chain Bring-up | src/Authority/StellaOps.Authority/TASKS.md | DONE (2025-10-23) | Authority Core & Security Guild | AUTH-MTLS-11-002 | Add OAuth mTLS client credential support with certificate-bound tokens and introspection updates. |
|
||||
| Sprint 12 | Runtime Guardrails | src/Scanner/StellaOps.Scanner.WebService/TASKS.md | DONE (2025-10-20) | Scanner WebService Guild | SCANNER-RUNTIME-12-301 | `/runtime/events` ingestion endpoint with validation, batching, storage hooks. |
|
||||
| Sprint 13 | UX & CLI Experience | src/Cli/StellaOps.Cli/TASKS.md | DONE (2025-10-21) | DevEx/CLI | CLI-OFFLINE-13-006 | Implement offline kit pull/import/status commands with integrity checks. |
|
||||
| Sprint 13 | UX & CLI Experience | src/Cli/StellaOps.Cli/TASKS.md | DONE (2025-10-22) | DevEx/CLI | CLI-PLUGIN-13-007 | Package non-core CLI verbs as restart-time plug-ins (manifest + loader tests). |
|
||||
| Sprint 13 | UX & CLI Experience | src/Web/StellaOps.Web/TASKS.md | DONE (2025-10-21) | UX Specialist, Angular Eng, DevEx | WEB1.DEPS-13-001 | Stabilise Angular workspace dependencies for headless CI installs (`npm install`, Chromium handling, docs). |
|
||||
| Sprint 16 | Scheduler Intelligence | src/Scheduler/__Libraries/StellaOps.Scheduler.Queue/TASKS.md | DONE (2025-10-20) | Scheduler Queue Guild | SCHED-QUEUE-16-403 | Dead-letter handling + metrics. |
|
||||
| Sprint 18 | Launch Readiness | ops/offline-kit/TASKS.md | DONE (2025-10-22) | Offline Kit Guild, Scanner Guild | DEVOPS-OFFLINE-18-004 | Rebuild Offline Kit bundle with Go analyzer plug-in and refreshed manifest/signature set. |
|
||||
34
docs/implplan/SPRINTS_PRIOR_20251025.md
Normal file
34
docs/implplan/SPRINTS_PRIOR_20251025.md
Normal file
@@ -0,0 +1,34 @@
|
||||
This file describe implementation of Stella Ops (docs/README.md). Implementation must respect rules from AGENTS.md (read if you have not).
|
||||
|
||||
| Sprint | Theme | Tasks File Path | Status | Type of Specialist | Task ID | Task Description |
|
||||
| --- | --- | --- | --- | --- | --- | --- |
|
||||
| Sprint 11 | Signing Chain Bring-up | src/Attestor/StellaOps.Attestor/TASKS.md | DONE (2025-10-19) | Attestor Guild | ATTESTOR-API-11-201 | `/rekor/entries` submission pipeline with dedupe, proof acquisition, and persistence. |
|
||||
| Sprint 11 | Signing Chain Bring-up | src/Attestor/StellaOps.Attestor/TASKS.md | DONE (2025-10-19) | Attestor Guild | ATTESTOR-VERIFY-11-202 | `/rekor/verify` + retrieval endpoints validating signatures and Merkle proofs. |
|
||||
| Sprint 11 | Signing Chain Bring-up | src/Attestor/StellaOps.Attestor/TASKS.md | DONE (2025-10-19) | Attestor Guild | ATTESTOR-OBS-11-203 | Telemetry, alerting, mTLS hardening, and archive workflow for Attestor. |
|
||||
| Sprint 11 | Storage Platform Hardening | src/Scanner/__Libraries/StellaOps.Scanner.Storage/TASKS.md | DONE (2025-10-23) | Scanner Storage Guild | SCANNER-STORAGE-11-401 | Migrate scanner object storage integration from MinIO to RustFS with data migration plan. |
|
||||
| Sprint 11 | UI Integration | src/UI/StellaOps.UI/TASKS.md | DONE (2025-10-23) | UI Guild | UI-ATTEST-11-005 | Attestation visibility (Rekor id, status) on Scan Detail. |
|
||||
| Sprint 12 | Runtime Guardrails | src/Zastava/__Libraries/StellaOps.Zastava.Core/TASKS.md | DONE (2025-10-23) | Zastava Core Guild | ZASTAVA-CORE-12-201 | Define runtime event/admission DTOs, hashing helpers, and versioning strategy. |
|
||||
| Sprint 12 | Runtime Guardrails | src/Zastava/__Libraries/StellaOps.Zastava.Core/TASKS.md | DONE (2025-10-23) | Zastava Core Guild | ZASTAVA-CORE-12-202 | Provide configuration/logging/metrics utilities shared by Observer/Webhook. |
|
||||
| Sprint 12 | Runtime Guardrails | src/Zastava/__Libraries/StellaOps.Zastava.Core/TASKS.md | DONE (2025-10-23) | Zastava Core Guild | ZASTAVA-CORE-12-203 | Authority client helpers, OpTok caching, and security guardrails for runtime services. |
|
||||
| Sprint 12 | Runtime Guardrails | src/Zastava/__Libraries/StellaOps.Zastava.Core/TASKS.md | DONE (2025-10-23) | Zastava Core Guild | ZASTAVA-OPS-12-204 | Operational runbooks, alert rules, and dashboard exports for runtime plane. |
|
||||
| Sprint 12 | Runtime Guardrails | src/Zastava/StellaOps.Zastava.Observer/TASKS.md | DONE (2025-10-24) | Zastava Observer Guild | ZASTAVA-OBS-12-001 | Container lifecycle watcher emitting deterministic runtime events with buffering. |
|
||||
| Sprint 12 | Runtime Guardrails | src/Zastava/StellaOps.Zastava.Observer/TASKS.md | DONE (2025-10-24) | Zastava Observer Guild | ZASTAVA-OBS-12-002 | Capture entrypoint traces + loaded libraries, hashing binaries and linking to baseline SBOM. |
|
||||
| Sprint 12 | Runtime Guardrails | src/Zastava/StellaOps.Zastava.Observer/TASKS.md | DONE (2025-10-24) | Zastava Observer Guild | ZASTAVA-OBS-12-003 | Posture checks for signatures/SBOM/attestation with offline caching. |
|
||||
| Sprint 12 | Runtime Guardrails | src/Zastava/StellaOps.Zastava.Observer/TASKS.md | DONE (2025-10-24) | Zastava Observer Guild | ZASTAVA-OBS-12-004 | Batch `/runtime/events` submissions with disk-backed buffer and rate limits. |
|
||||
| Sprint 12 | Runtime Guardrails | src/Zastava/StellaOps.Zastava.Webhook/TASKS.md | DONE (2025-10-24) | Zastava Webhook Guild | ZASTAVA-WEBHOOK-12-101 | Admission controller host with TLS bootstrap and Authority auth. |
|
||||
| Sprint 12 | Runtime Guardrails | src/Zastava/StellaOps.Zastava.Webhook/TASKS.md | DONE (2025-10-24) | Zastava Webhook Guild | ZASTAVA-WEBHOOK-12-102 | Query Scanner `/policy/runtime`, resolve digests, enforce verdicts. |
|
||||
| Sprint 12 | Runtime Guardrails | src/Zastava/StellaOps.Zastava.Webhook/TASKS.md | DONE (2025-10-24) | Zastava Webhook Guild | ZASTAVA-WEBHOOK-12-103 | Caching, fail-open/closed toggles, metrics/logging for admission decisions. |
|
||||
| Sprint 12 | Runtime Guardrails | src/Zastava/StellaOps.Zastava.Webhook/TASKS.md | DONE (2025-10-24) | Zastava Webhook Guild | ZASTAVA-WEBHOOK-12-104 | Wire `/admission` endpoint to runtime policy client and emit allow/deny envelopes. |
|
||||
| Sprint 12 | Runtime Guardrails | src/Scanner/StellaOps.Scanner.WebService/TASKS.md | DONE (2025-10-24) | Scanner WebService Guild | SCANNER-RUNTIME-12-302 | `/policy/runtime` endpoint joining SBOM baseline + policy verdict, returning admission guidance. |
|
||||
| Sprint 12 | Runtime Guardrails | src/Scanner/StellaOps.Scanner.WebService/TASKS.md | DONE (2025-10-24) | Scanner WebService Guild | SCANNER-RUNTIME-12-303 | Align `/policy/runtime` verdicts with canonical policy evaluation (Feedser/Vexer). |
|
||||
| Sprint 12 | Runtime Guardrails | src/Scanner/StellaOps.Scanner.WebService/TASKS.md | DONE (2025-10-24) | Scanner WebService Guild | SCANNER-RUNTIME-12-304 | Integrate attestation verification into runtime policy metadata. |
|
||||
| Sprint 12 | Runtime Guardrails | src/Scanner/StellaOps.Scanner.WebService/TASKS.md | DONE (2025-10-24) | Scanner WebService Guild | SCANNER-RUNTIME-12-305 | Deliver shared fixtures + e2e validation with Zastava/CLI teams. |
|
||||
| Sprint 13 | UX & CLI Experience | src/UI/StellaOps.UI/TASKS.md | DONE (2025-10-23) | UI Guild | UI-AUTH-13-001 | Integrate Authority OIDC + DPoP flows with session management. |
|
||||
| Sprint 13 | UX & CLI Experience | src/UI/StellaOps.UI/TASKS.md | DONE (2025-10-25) | UI Guild | UI-NOTIFY-13-006 | Notify panel: channels/rules CRUD, deliveries view, test send. |
|
||||
| Sprint 13 | Platform Reliability | ops/devops/TASKS.md | DONE (2025-10-25) | DevOps Guild, Platform Leads | DEVOPS-NUGET-13-001 | Wire up .NET 10 preview feeds/local mirrors so `dotnet restore` succeeds offline; document updated NuGet bootstrap. |
|
||||
| Sprint 15 | Notify Foundations | src/Notify/__Libraries/StellaOps.Notify.Queue/TASKS.md | DONE (2025-10-23) | Notify Queue Guild | NOTIFY-QUEUE-15-401 | Bus abstraction + Redis Streams adapter with ordering/idempotency. |
|
||||
| Sprint 15 | Notify Foundations | src/Notify/__Libraries/StellaOps.Notify.Queue/TASKS.md | DONE (2025-10-23) | Notify Queue Guild | NOTIFY-QUEUE-15-402 | NATS JetStream adapter with health probes and failover. |
|
||||
| Sprint 15 | Notify Foundations | src/Notify/__Libraries/StellaOps.Notify.Queue/TASKS.md | DONE (2025-10-23) | Notify Queue Guild | NOTIFY-QUEUE-15-403 | Delivery queue with retry/dead-letter + metrics. |
|
||||
| Sprint 15 | Notify Foundations | src/Notify/StellaOps.Notify.Worker/TASKS.md | DONE (2025-10-23) | Notify Worker Guild | NOTIFY-WORKER-15-201 | Bus subscription + leasing loop with backoff. |
|
||||
| Sprint 17 | Symbol Intelligence & Forensics | src/Zastava/StellaOps.Zastava.Observer/TASKS.md | DONE (2025-10-25) | Zastava Observer Guild | ZASTAVA-OBS-17-005 | Collect GNU build-id during runtime observation and attach it to emitted events. |
|
||||
| Sprint 17 | Symbol Intelligence & Forensics | src/Scanner/StellaOps.Scanner.WebService/TASKS.md | DONE (2025-10-25) | Scanner WebService Guild | SCANNER-RUNTIME-17-401 | Persist runtime build-id observations and expose them for debug-symbol correlation. |
|
||||
84
docs/implplan/SPRINTS_PRIOR_20251027.md
Normal file
84
docs/implplan/SPRINTS_PRIOR_20251027.md
Normal file
@@ -0,0 +1,84 @@
|
||||
This file describe implementation of Stella Ops (docs/README.md). Implementation must respect rules from AGENTS.md (read if you have not).
|
||||
|
||||
| Sprint | Theme | Tasks File Path | Status | Type of Specialist | Task ID | Task Description |
|
||||
| --- | --- | --- | --- | --- | --- | --- |
|
||||
| Sprint 13 | Platform Reliability | ops/devops/TASKS.md | DONE (2025-10-26) | DevOps Guild | DEVOPS-NUGET-13-002 | Ensure all solutions/projects prioritize `local-nuget` before public feeds and add restore-order validation. |
|
||||
| Sprint 13 | Platform Reliability | ops/devops/TASKS.md | DONE (2025-10-26) | DevOps Guild, Platform Leads | DEVOPS-NUGET-13-003 | Upgrade `Microsoft.*` dependencies pinned to 8.* to their latest .NET 10 (or 9.x) releases and refresh guidance. |
|
||||
| Sprint 14 | Release & Offline Ops | ops/deployment/TASKS.md | DONE (2025-10-26) | Deployment Guild | DEVOPS-OPS-14-003 | Deployment/update/rollback automation and channel management documentation. |
|
||||
| Sprint 14 | Release & Offline Ops | ops/devops/TASKS.md | DONE (2025-10-26) | DevOps Guild | DEVOPS-REL-14-001 | Deterministic build/release pipeline with SBOM/provenance, signing, and manifest generation. |
|
||||
| Sprint 14 | Release & Offline Ops | ops/devops/TASKS.md | DONE (2025-10-26) | DevOps Guild, Scanner Guild | DEVOPS-REL-14-004 | Extend release/offline smoke jobs to cover Python analyzer plug-ins (warm/cold, determinism, signing). |
|
||||
| Sprint 14 | Release & Offline Ops | ops/licensing/TASKS.md | DONE (2025-10-26) | Licensing Guild | DEVOPS-LIC-14-004 | Registry token service tied to Authority, plan gating, revocation handling, monitoring. |
|
||||
| Sprint 14 | Release & Offline Ops | ops/offline-kit/TASKS.md | DONE (2025-10-26) | Offline Kit Guild | DEVOPS-OFFLINE-14-002 | Offline kit packaging workflow with integrity verification and documentation. |
|
||||
| Sprint 15 | Benchmarks | src/Bench/StellaOps.Bench/TASKS.md | DONE (2025-10-26) | Bench Guild, Notify Team | BENCH-NOTIFY-15-001 | Notify dispatch throughput bench with results CSV. |
|
||||
| Sprint 16 | Scheduler Intelligence | src/Scheduler/__Libraries/StellaOps.Scheduler.Models/TASKS.md | DONE (2025-10-19) | Scheduler Models Guild | SCHED-MODELS-16-101 | Define Scheduler DTOs & validation. |
|
||||
| Sprint 16 | Scheduler Intelligence | src/Scheduler/__Libraries/StellaOps.Scheduler.Models/TASKS.md | DONE (2025-10-19) | Scheduler Models Guild | SCHED-MODELS-16-102 | Publish schema docs/sample payloads. |
|
||||
| Sprint 16 | Scheduler Intelligence | src/Scheduler/__Libraries/StellaOps.Scheduler.Storage.Mongo/TASKS.md | DONE (2025-10-19) | Scheduler Storage Guild | SCHED-STORAGE-16-201 | Mongo schemas/indexes for Scheduler state. |
|
||||
| Sprint 16 | Scheduler Intelligence | src/Scheduler/__Libraries/StellaOps.Scheduler.Storage.Mongo/TASKS.md | DONE (2025-10-26) | Scheduler Storage Guild | SCHED-STORAGE-16-202 | Repositories with tenant scoping, TTL, causal consistency. |
|
||||
| Sprint 16 | Scheduler Intelligence | src/Scheduler/__Libraries/StellaOps.Scheduler.Storage.Mongo/TASKS.md | DONE (2025-10-26) | Scheduler Storage Guild | SCHED-STORAGE-16-203 | Audit/run stats materialization for UI. |
|
||||
| Sprint 16 | Scheduler Intelligence | src/Scheduler/__Libraries/StellaOps.Scheduler.ImpactIndex/TASKS.md | DONE (2025-10-26) | Scheduler ImpactIndex Guild | SCHED-IMPACT-16-302 | Query APIs for ResolveByPurls/ResolveByVulns/ResolveAll. |
|
||||
| Sprint 16 | Scheduler Intelligence | src/Scheduler/__Libraries/StellaOps.Scheduler.ImpactIndex/TASKS.md | DONE (2025-10-26) | Scheduler ImpactIndex Guild | SCHED-IMPACT-16-301 | Ingest BOM-Index into roaring bitmap store. |
|
||||
| Sprint 16 | Scheduler Intelligence | src/Scheduler/StellaOps.Scheduler.WebService/TASKS.md | DONE (2025-10-26) | Scheduler WebService Guild | SCHED-WEB-16-102 | Schedules CRUD (cron validation, pause/resume, audit). |
|
||||
| Sprint 16 | Scheduler Intelligence | src/Scheduler/StellaOps.Scheduler.WebService/TASKS.md | DONE (2025-10-26) | Scheduler WebService Guild | SCHED-WEB-16-103 | Runs API (list/detail/cancel) + impact previews. |
|
||||
| Sprint 16 | Scheduler Intelligence | src/Scheduler/StellaOps.Scheduler.WebService/TASKS.md | DONE (2025-10-27) | Scheduler WebService Guild | SCHED-WEB-16-104 | Feedser/Vexer webhook handlers with security enforcement. |
|
||||
| Sprint 17 | Symbol Intelligence & Forensics | docs/TASKS.md | DONE (2025-10-26) | Docs Guild | DOCS-RUNTIME-17-004 | Document build-id workflows for SBOMs, runtime events, and debug-store usage. |
|
||||
| Sprint 17 | Symbol Intelligence & Forensics | ops/devops/TASKS.md | DONE (2025-10-26) | DevOps Guild | DEVOPS-REL-17-002 | Ship stripped debug artifacts organised by build-id within release/offline kits. |
|
||||
| Sprint 17 | Symbol Intelligence & Forensics | ops/offline-kit/TASKS.md | DONE (2025-10-26) | Offline Kit Guild, DevOps Guild | DEVOPS-OFFLINE-17-003 | Mirror release debug-store artefacts into Offline Kit packaging and document validation. |
|
||||
| Sprint 17 | Symbol Intelligence & Forensics | src/Scanner/__Libraries/StellaOps.Scanner.Emit/TASKS.md | DONE (2025-10-26) | Emit Guild | SCANNER-EMIT-17-701 | Record GNU build-id for ELF components and surface it in SBOM/diff outputs. |
|
||||
| Sprint 18 | Launch Readiness | ops/devops/TASKS.md | DONE (2025-10-26) | DevOps Guild | DEVOPS-LAUNCH-18-001 | Production launch cutover rehearsal and runbook publication. |
|
||||
| Sprint 18 | Launch Readiness | ops/offline-kit/TASKS.md | DONE (2025-10-26) | Offline Kit Guild, Scanner Guild | DEVOPS-OFFLINE-18-005 | Rebuild Offline Kit with Python analyzer artefacts and refreshed manifest/signature pair. |
|
||||
| Sprint 19 | Aggregation-Only Contract Enforcement | docs/TASKS.md | DONE (2025-10-26) | Docs Guild | DOCS-AOC-19-001 | Publish aggregation-only contract reference documentation. |
|
||||
| Sprint 19 | Aggregation-Only Contract Enforcement | docs/TASKS.md | DONE (2025-10-26) | Docs Guild, Architecture Guild | DOCS-AOC-19-002 | Update architecture overview with AOC boundary diagrams. |
|
||||
| Sprint 19 | Aggregation-Only Contract Enforcement | docs/TASKS.md | DONE (2025-10-26) | Docs Guild, Policy Guild | DOCS-AOC-19-003 | Refresh policy engine doc with raw ingestion constraints. |
|
||||
| Sprint 19 | Aggregation-Only Contract Enforcement | docs/TASKS.md | DONE (2025-10-26) | Docs Guild, UI Guild | DOCS-AOC-19-004 | Document console AOC dashboard and drill-down flow. |
|
||||
| Sprint 19 | Aggregation-Only Contract Enforcement | docs/TASKS.md | DONE (2025-10-26) | Docs Guild, CLI Guild | DOCS-AOC-19-005 | Document CLI AOC commands and exit codes. |
|
||||
| Sprint 19 | Aggregation-Only Contract Enforcement | docs/TASKS.md | DONE (2025-10-26) | Docs Guild, Observability Guild | DOCS-AOC-19-006 | Document new AOC metrics, traces, and logs. |
|
||||
| Sprint 19 | Aggregation-Only Contract Enforcement | docs/TASKS.md | DONE (2025-10-26) | Docs Guild, Authority Core | DOCS-AOC-19-007 | Document new Authority scopes and tenancy enforcement. |
|
||||
| Sprint 19 | Aggregation-Only Contract Enforcement | docs/TASKS.md | DONE (2025-10-26) | Docs Guild, DevOps Guild | DOCS-AOC-19-008 | Update deployment guide with validator enablement and verify user guidance. |
|
||||
| Sprint 19 | Aggregation-Only Contract Enforcement | src/Authority/StellaOps.Authority/TASKS.md | DONE (2025-10-26) | Authority Core & Security Guild | AUTH-AOC-19-001 | Introduce new ingestion/auth scopes across Authority. |
|
||||
| Sprint 20 | Policy Engine v2 | docs/TASKS.md | DONE (2025-10-26) | Docs Guild | DOCS-POLICY-20-001 | Publish `/docs/policy/overview.md` with compliance checklist. |
|
||||
| Sprint 20 | Policy Engine v2 | docs/TASKS.md | DONE (2025-10-26) | Docs Guild | DOCS-POLICY-20-002 | Document DSL grammar + examples in `/docs/policy/dsl.md`. |
|
||||
| Sprint 20 | Policy Engine v2 | docs/TASKS.md | DONE (2025-10-26) | Docs Guild, Authority Core | DOCS-POLICY-20-003 | Write `/docs/policy/lifecycle.md` covering workflow + roles. |
|
||||
| Sprint 20 | Policy Engine v2 | docs/TASKS.md | DONE (2025-10-26) | Docs Guild, Scheduler Guild | DOCS-POLICY-20-004 | Document policy run modes + cursors in `/docs/policy/runs.md`. |
|
||||
| Sprint 20 | Policy Engine v2 | docs/TASKS.md | DONE (2025-10-26) | Docs Guild, Platform Guild | DOCS-POLICY-20-005 | Produce `/docs/api/policy.md` with endpoint schemas + errors. |
|
||||
| Sprint 20 | Policy Engine v2 | docs/TASKS.md | DONE (2025-10-26) | Docs Guild, CLI Guild | DOCS-POLICY-20-006 | Author `/docs/cli/policy.md` with commands, exit codes, JSON output. |
|
||||
| Sprint 20 | Policy Engine v2 | docs/TASKS.md | DONE (2025-10-26) | Docs Guild, UI Guild | DOCS-POLICY-20-007 | Create `/docs/ui/policy-editor.md` covering editor, simulation, approvals. |
|
||||
| Sprint 20 | Policy Engine v2 | docs/TASKS.md | DONE (2025-10-26) | Docs Guild, Architecture Guild | DOCS-POLICY-20-008 | Publish `/docs/architecture/policy-engine.md` with sequence diagrams. |
|
||||
| Sprint 20 | Policy Engine v2 | docs/TASKS.md | DONE (2025-10-26) | Docs Guild, Observability Guild | DOCS-POLICY-20-009 | Document metrics/traces/logs in `/docs/observability/policy.md`. |
|
||||
| Sprint 20 | Policy Engine v2 | docs/TASKS.md | DONE (2025-10-26) | Docs Guild, Security Guild | DOCS-POLICY-20-010 | Publish `/docs/security/policy-governance.md` for scopes + approvals. |
|
||||
| Sprint 20 | Policy Engine v2 | docs/TASKS.md | DONE (2025-10-26) | Docs Guild, Policy Guild | DOCS-POLICY-20-011 | Add example policies under `/docs/examples/policies/` with commentary. |
|
||||
| Sprint 20 | Policy Engine v2 | docs/TASKS.md | DONE (2025-10-26) | Docs Guild, Support Guild | DOCS-POLICY-20-012 | Draft `/docs/faq/policy-faq.md` covering conflicts, determinism, pitfalls. |
|
||||
| Sprint 20 | Policy Engine v2 | ops/devops/TASKS.md | DONE (2025-10-26) | DevOps Guild | DEVOPS-POLICY-20-001 | Add DSL lint + compile checks to CI pipelines. |
|
||||
| Sprint 20 | Policy Engine v2 | ops/devops/TASKS.md | DONE (2025-10-26) | DevOps Guild, QA Guild | DEVOPS-POLICY-20-003 | Add determinism CI job diffing repeated policy runs. |
|
||||
| Sprint 20 | Policy Engine v2 | samples/TASKS.md | DONE (2025-10-26) | Samples Guild, Policy Guild | SAMPLES-POLICY-20-001 | Commit baseline/serverless/internal-only policy samples + fixtures. |
|
||||
| Sprint 20 | Policy Engine v2 | samples/TASKS.md | DONE (2025-10-26) | Samples Guild, UI Guild | SAMPLES-POLICY-20-002 | Produce simulation diff fixtures for UI/CLI tests. |
|
||||
| Sprint 20 | Policy Engine v2 | src/Authority/StellaOps.Authority/TASKS.md | DONE (2025-10-26) | Authority Core & Security Guild | AUTH-POLICY-20-001 | Add new policy scopes (`policy:*`, `findings:read`, `effective:write`). |
|
||||
| Sprint 20 | Policy Engine v2 | src/Authority/StellaOps.Authority/TASKS.md | DONE (2025-10-26) | Authority Core & Security Guild | AUTH-POLICY-20-002 | Enforce Policy Engine service identity and scope checks at gateway. |
|
||||
| Sprint 20 | Policy Engine v2 | src/Authority/StellaOps.Authority/TASKS.md | DONE (2025-10-26) | Authority Core & Docs Guild | AUTH-POLICY-20-003 | Update Authority docs/config samples for policy scopes + workflows. |
|
||||
| Sprint 20 | Policy Engine v2 | src/Bench/StellaOps.Bench/TASKS.md | DONE (2025-10-26) | Bench Guild, Policy Guild | BENCH-POLICY-20-001 | Create policy evaluation benchmark suite + baseline metrics. |
|
||||
| Sprint 20 | Policy Engine v2 | src/Policy/StellaOps.Policy.Engine/TASKS.md | DONE (2025-10-26) | Policy Guild, Platform Guild | POLICY-ENGINE-20-000 | Spin up new Policy Engine service host with DI bootstrap and Authority wiring. |
|
||||
| Sprint 20 | Policy Engine v2 | src/Policy/StellaOps.Policy.Engine/TASKS.md | DONE (2025-10-26) | Policy Guild | POLICY-ENGINE-20-001 | Deliver `stella-dsl@1` parser + IR compiler with diagnostics and checksums. |
|
||||
| Sprint 20 | Policy Engine v2 | src/Scheduler/__Libraries/StellaOps.Scheduler.Models/TASKS.md | DONE (2025-10-26) | Scheduler Models Guild | SCHED-MODELS-20-001 | Define policy run/diff DTOs + validation helpers. |
|
||||
| Sprint 21 | Graph Explorer v1 | src/Authority/StellaOps.Authority/TASKS.md | DONE (2025-10-26) | Authority Core Guild | AUTH-GRAPH-21-001 | Introduce graph scopes (`graph:*`) with configuration binding and defaults. |
|
||||
| Sprint 21 | Graph Explorer v1 | src/Authority/StellaOps.Authority/TASKS.md | DONE (2025-10-26) | Authority Core Guild | AUTH-GRAPH-21-002 | Enforce graph scopes/identities at gateway with tenant propagation. |
|
||||
| Sprint 21 | Graph Explorer v1 | src/Authority/StellaOps.Authority/TASKS.md | DONE (2025-10-26) | Authority Core & Docs Guild | AUTH-GRAPH-21-003 | Update security docs/config samples for graph access and least privilege. |
|
||||
| Sprint 21 | Graph Explorer v1 | src/Scheduler/__Libraries/StellaOps.Scheduler.Models/TASKS.md | DONE (2025-10-26) | Scheduler Models Guild | SCHED-MODELS-21-001 | Define job DTOs for graph builds/overlay refresh (`GraphBuildJob`, `GraphOverlayJob`) with deterministic serialization and status enums; document in `src/Scheduler/__Libraries/StellaOps.Scheduler.Models/docs/SCHED-MODELS-21-001-GRAPH-JOBS.md`. |
|
||||
| Sprint 21 | Graph Explorer v1 | src/Scheduler/__Libraries/StellaOps.Scheduler.Models/TASKS.md | DONE (2025-10-26) | Scheduler Models Guild | SCHED-MODELS-21-002 | Publish schema docs/sample payloads for graph job lifecycle. |
|
||||
| Sprint 22 | Link-Not-Merge v1 | src/Bench/StellaOps.Bench/TASKS.md | DONE (2025-10-26) | Bench Guild | BENCH-LNM-22-001 | Benchmark advisory observation ingest/correlation throughput. |
|
||||
| Sprint 22 | Link-Not-Merge v1 | src/Bench/StellaOps.Bench/TASKS.md | DONE (2025-10-26) | Bench Guild | BENCH-LNM-22-002 | Benchmark VEX ingest/correlation latency and event emission. |
|
||||
| Sprint 23 | StellaOps Console | docs/TASKS.md | DONE (2025-10-26) | Docs Guild | DOCS-CONSOLE-23-001 | Publish `/docs/ui/console-overview.md` (IA, tenant model, filters, AOC alignment). |
|
||||
| Sprint 23 | StellaOps Console | docs/TASKS.md | DONE (2025-10-26) | Docs Guild | DOCS-CONSOLE-23-002 | Author `/docs/ui/navigation.md` with route map, filters, keyboard shortcuts, deep links. |
|
||||
| Sprint 23 | StellaOps Console | docs/TASKS.md | DONE (2025-10-26) | Docs Guild | DOCS-CONSOLE-23-003 | Document `/docs/ui/sbom-explorer.md` covering catalog, graph, overlays, exports. |
|
||||
| Sprint 23 | StellaOps Console | docs/TASKS.md | DONE (2025-10-26) | Docs Guild | DOCS-CONSOLE-23-004 | Produce `/docs/ui/advisories-and-vex.md` detailing aggregation-not-merge UX. |
|
||||
| Sprint 23 | StellaOps Console | docs/TASKS.md | DONE (2025-10-26) | Docs Guild | DOCS-CONSOLE-23-005 | Write `/docs/ui/findings.md` with filters, explain, exports, CLI parity notes. |
|
||||
| Sprint 23 | StellaOps Console | docs/TASKS.md | DONE (2025-10-26) | Docs Guild | DOCS-CONSOLE-23-006 | Publish `/docs/ui/policies.md` (editor, simulation, approvals, RBAC). |
|
||||
| Sprint 23 | StellaOps Console | docs/TASKS.md | DONE (2025-10-26) | Docs Guild | DOCS-CONSOLE-23-007 | Document `/docs/ui/runs.md` with SSE monitoring, diff, retries, evidence downloads. |
|
||||
| Sprint 23 | StellaOps Console | docs/TASKS.md | DONE (2025-10-26) | Docs Guild | DOCS-CONSOLE-23-008 | Draft `/docs/ui/admin.md` covering tenants, roles, tokens, integrations, fresh-auth. |
|
||||
| Sprint 23 | StellaOps Console | docs/TASKS.md | DONE (2025-10-27) | Docs Guild | DOCS-CONSOLE-23-009 | Publish `/docs/ui/downloads.md` aligning manifest with commands and offline flow. |
|
||||
| Sprint 23 | StellaOps Console | docs/TASKS.md | DONE (2025-10-27) | Docs Guild, Deployment Guild, Console Guild | DOCS-CONSOLE-23-010 | Write `/docs/deploy/console.md` (Helm, ingress, TLS, env vars, health checks). |
|
||||
| Sprint 28 | Graph Explorer | src/Scheduler/StellaOps.Scheduler.WebService/TASKS.md | DONE (2025-10-26) | Scheduler WebService Guild | SCHED-WEB-21-001 | Provide graph build/overlay job APIs; see `docs/SCHED-WEB-21-001-GRAPH-APIS.md`. |
|
||||
| Sprint 28 | Graph Explorer | src/Scheduler/StellaOps.Scheduler.WebService/TASKS.md | DONE (2025-10-26) | Scheduler WebService Guild | SCHED-WEB-21-002 | Provide overlay lag metrics endpoint/webhook; see `docs/SCHED-WEB-21-001-GRAPH-APIS.md`. |
|
||||
| Sprint 28 | Graph Explorer | src/Scheduler/StellaOps.Scheduler.WebService/TASKS.md | DONE (2025-10-26) | Scheduler WebService Guild, Authority Core Guild | SCHED-WEB-21-003 | Replace header auth with Authority scopes using `StellaOpsScopes`; dev fallback only when `Scheduler:Authority:Enabled=false`. |
|
||||
| Sprint 50 | Observability & Forensics Phase 1 – Baseline Telemetry | ops/devops/TASKS.md | DONE (2025-10-26) | DevOps Guild | DEVOPS-OBS-50-001 | Deploy default OpenTelemetry collector manifests with secure OTLP pipeline. |
|
||||
| Sprint 50 | Observability & Forensics Phase 1 – Baseline Telemetry | ops/devops/TASKS.md | DONE (2025-10-26) | DevOps Guild | DEVOPS-OBS-50-003 | Package telemetry stack configs for offline/air-gapped installs with signatures. |
|
||||
| Sprint 16 | Scheduler Intelligence | src/Scheduler/StellaOps.Scheduler.WebService/TASKS.md | DONE (2025-10-27) | Scheduler WebService Guild | SCHED-WEB-16-101 | Minimal API host with Authority enforcement. |
|
||||
| Sprint 16 | Scheduler Intelligence | src/Scheduler/__Libraries/StellaOps.Scheduler.Worker/TASKS.md | DONE (2025-10-27) | Scheduler Worker Guild | SCHED-WORKER-16-202 | ImpactIndex targeting and shard planning. |
|
||||
26
docs/implplan/SPRINTS_PRIOR_20251028.md
Normal file
26
docs/implplan/SPRINTS_PRIOR_20251028.md
Normal file
@@ -0,0 +1,26 @@
|
||||
This file describe implementation of Stella Ops (docs/README.md). Implementation must respect rules from AGENTS.md (read if you have not).
|
||||
|
||||
| Sprint | Theme | Tasks File Path | Status | Type of Specialist | Task ID | Task Description |
|
||||
| --- | --- | --- | --- | --- | --- | --- |
|
||||
| Sprint 16 | Scheduler Intelligence | src/Scheduler/__Libraries/StellaOps.Scheduler.Worker/TASKS.md | DONE (2025-10-27) | Scheduler Worker Guild | SCHED-WORKER-16-203 | Runner execution invoking Scanner analysis/content refresh. |
|
||||
| Sprint 16 | Scheduler Intelligence | src/Scheduler/__Libraries/StellaOps.Scheduler.Worker/TASKS.md | DONE (2025-10-27) | Scheduler Worker Guild | SCHED-WORKER-16-204 | Emit rescan/report events for Notify/UI. |
|
||||
| Sprint 16 | Scheduler Intelligence | src/Scheduler/__Libraries/StellaOps.Scheduler.Worker/TASKS.md | DONE (2025-10-27) | Scheduler Worker Guild | SCHED-WORKER-16-205 | Metrics/telemetry for Scheduler planners/runners. |
|
||||
| Sprint 19 | Aggregation-Only Contract Enforcement | src/Authority/StellaOps.Authority/TASKS.md | DONE (2025-10-27) | Authority Core & Security Guild | AUTH-AOC-19-002 | Enforce tenant claim propagation and cross-tenant guardrails. |
|
||||
> AUTH-AOC-19-002: Tenant metadata now flows through rate limiter/audit/token persistence; password grant scope/tenant enforcement landed. Docs/stakeholder walkthrough pending.
|
||||
> 2025-10-27 Update: Ingestion scopes require tenant assignment; access tokens propagate tenant claims and reject cross-tenant mismatches with coverage.
|
||||
| Sprint 19 | Aggregation-Only Contract Enforcement | src/Authority/StellaOps.Authority/TASKS.md | DONE (2025-10-27) | Authority Core & Docs Guild | AUTH-AOC-19-003 | Update Authority docs/config samples for new scopes. |
|
||||
> AUTH-AOC-19-003: Scope catalogue, console/CLI docs, and sample config updated to require `aoc:verify` plus read scopes; verification clients now explicitly include tenant hints. Authority test run remains blocked on Concelier build failure (`ImmutableHashSet<string?>`), previously noted under AUTH-AOC-19-002.
|
||||
| Sprint 19 | Aggregation-Only Contract Enforcement | src/Concelier/StellaOps.Concelier.WebService/TASKS.md | DONE (2025-10-28) | Concelier WebService Guild | CONCELIER-WEB-AOC-19-001 | Implement raw advisory ingestion endpoints with AOC guard and verifier. |
|
||||
| Sprint 19 | Aggregation-Only Contract Enforcement | src/Excititor/StellaOps.Excititor.Worker/TASKS.md | DONE (2025-10-28) | QA Guild | EXCITITOR-WORKER-AOC-19-003 | Expand worker tests for deterministic batching and restart safety. |
|
||||
| Sprint 20 | Policy Engine v2 | ops/devops/TASKS.md | DONE (2025-10-27) | DevOps Guild, Scheduler Guild, CLI Guild | DEVOPS-POLICY-20-004 | Automate policy schema exports and change notifications for CLI consumers. |
|
||||
| Sprint 20 | Policy Engine v2 | src/Cli/StellaOps.Cli/TASKS.md | DONE (2025-10-27) | DevEx/CLI Guild | CLI-POLICY-20-002 | Implement `stella policy simulate` with diff outputs + exit codes. |
|
||||
| Sprint 21 | Graph Explorer v1 | src/Cartographer/StellaOps.Cartographer/TASKS.md | DONE (2025-10-27) | Cartographer Guild | CARTO-GRAPH-21-010 | Replace hard-coded `graph:*` scope strings with shared constants once graph services integrate. |
|
||||
| Sprint 21 | Graph Explorer v1 | src/Scheduler/StellaOps.Scheduler.WebService/TASKS.md | DONE (2025-10-26) | Scheduler WebService Guild | SCHED-WEB-21-002 | Expose overlay lag metrics and job completion hooks for Cartographer. |
|
||||
| Sprint 23 | StellaOps Console | docs/TASKS.md | DONE (2025-10-28) | Docs Guild | DOCS-CONSOLE-23-011 | Update `/docs/install/docker.md` to include console image, compose/Helm/offline examples. |
|
||||
| Sprint 23 | StellaOps Console | docs/TASKS.md | DONE (2025-10-28) | Docs Guild | DOCS-CONSOLE-23-012 | Publish `/docs/security/console-security.md` covering OIDC, scopes, CSP, evidence handling. |
|
||||
| Sprint 23 | StellaOps Console | docs/TASKS.md | DONE (2025-10-28) | Docs Guild | DOCS-CONSOLE-23-013 | Write `/docs/observability/ui-telemetry.md` cataloguing metrics/logs/dashboards/alerts. |
|
||||
| Sprint 23 | StellaOps Console | docs/TASKS.md | DONE (2025-10-28) | Docs Guild | DOCS-CONSOLE-23-014 | Maintain `/docs/cli-vs-ui-parity.md` matrix with CI drift detection guidance. |
|
||||
| Sprint 23 | StellaOps Console | docs/TASKS.md | DONE (2025-10-28) | Docs Guild | DOCS-CONSOLE-23-016 | Refresh `/docs/accessibility.md` with console keyboard flows, tokens, testing tools. <br>2025-10-28: Published guide covering keyboard matrix, screen-reader behaviour, colour tokens, testing workflow, offline guidance, and compliance checklist. |
|
||||
| Sprint 25 | Exceptions v1 | docs/TASKS.md | DONE (2025-10-27) | Docs Guild | DOCS-EXC-25-004 | Document policy exception effects + simulation. |
|
||||
| Sprint 25 | Exceptions v1 | src/Policy/StellaOps.Policy.Engine/TASKS.md | DONE (2025-10-27) | Policy Guild | POLICY-ENGINE-70-001 | Add exception evaluation layer with specificity + effects. |
|
||||
| Sprint 25 | Exceptions v1 | src/Policy/__Libraries/StellaOps.Policy/TASKS.md | DONE (2025-10-27) | Policy Guild | POLICY-EXC-25-001 | Extend SPL schema to reference exception effects and routing. |
|
||||
@@ -1,180 +1,180 @@
|
||||
# Aggregation-Only Contract Reference
|
||||
|
||||
> The Aggregation-Only Contract (AOC) is the governing rule set that keeps StellaOps ingestion services deterministic, policy-neutral, and auditable. It applies to Concelier, Excititor, and any future collectors that write raw advisory or VEX documents.
|
||||
|
||||
## 1. Purpose and Scope
|
||||
|
||||
- Defines the canonical behaviour for `advisory_raw` and `vex_raw` collections and the linkset hints they may emit.
|
||||
- Applies to every ingestion runtime (`StellaOps.Concelier.*`, `StellaOps.Excititor.*`), the Authority scopes that guard them, and the DevOps/QA surfaces that verify compliance.
|
||||
- Complements the high-level architecture in [Concelier](../ARCHITECTURE_CONCELIER.md) and Authority enforcement documented in [Authority Architecture](../ARCHITECTURE_AUTHORITY.md).
|
||||
- Paired guidance: see the guard-rail checkpoints in [AOC Guardrails](../aoc/aoc-guardrails.md) and CLI usage that will land in `/docs/cli/` as part of Sprint 19 follow-up.
|
||||
|
||||
## 2. Philosophy and Goals
|
||||
|
||||
- Preserve upstream truth: ingestion only captures immutable raw facts plus provenance, never derived severity or policy decisions.
|
||||
- Defer interpretation: Policy Engine and downstream overlays remain the sole writers of materialised findings, severity, consensus, or risk scores.
|
||||
- Make every write explainable: provenance, signatures, and content hashes are required so operators can prove where each fact originated.
|
||||
- Keep outputs reproducible: identical inputs must yield identical documents, hashes, and linksets across replays and air-gapped installs.
|
||||
|
||||
## 3. Contract Invariants
|
||||
|
||||
| # | Invariant | What it forbids or requires | Enforcement surfaces |
|
||||
|---|-----------|-----------------------------|----------------------|
|
||||
| 1 | No derived severity at ingest | Reject top-level keys such as `severity`, `cvss`, `effective_status`, `consensus_provider`, `risk_score`. Raw upstream CVSS remains inside `content.raw`. | Mongo schema validator, `AOCWriteGuard`, Roslyn analyzer, `stella aoc verify`. |
|
||||
| 2 | No merges or opinionated dedupe | Each upstream document persists on its own; ingestion never collapses multiple vendors into one document. | Repository interceptors, unit/fixture suites. |
|
||||
| 3 | Provenance is mandatory | `source.*`, `upstream.*`, and `signature` metadata must be present; missing provenance triggers `ERR_AOC_004`. | Schema validator, guard, CLI verifier. |
|
||||
| 4 | Idempotent upserts | Writes keyed by `(vendor, upstream_id, content_hash)` either no-op or insert a new revision with `supersedes`. Duplicate hashes map to the same document. | Repository guard, storage unique index, CI smoke tests. |
|
||||
| 5 | Append-only revisions | Updates create a new document with `supersedes` pointer; no in-place mutation of content. | Mongo schema (`supersedes` format), guard, data migration scripts. |
|
||||
| 6 | Linkset only | Ingestion may compute link hints (`purls`, `cpes`, IDs) to accelerate joins, but must not transform or infer severity or policy. | Linkset builders reviewed via fixtures and analyzers. |
|
||||
| 7 | Policy-only effective findings | Only Policy Engine identities can write `effective_finding_*`; ingestion callers receive `ERR_AOC_006` if they attempt it. | Authority scopes, Policy Engine guard. |
|
||||
| 8 | Schema safety | Unknown top-level keys reject with `ERR_AOC_007`; timestamps use ISO 8601 UTC strings; tenant is required. | Mongo validator, JSON schema tests. |
|
||||
| 9 | Clock discipline | Collectors stamp `fetched_at` and `received_at` monotonically per batch to support reproducibility windows. | Collector contracts, QA fixtures. |
|
||||
|
||||
## 4. Raw Schemas
|
||||
|
||||
### 4.1 `advisory_raw`
|
||||
|
||||
| Field | Type | Notes |
|
||||
|-------|------|-------|
|
||||
| `_id` | string | `advisory_raw:{source}:{upstream_id}:{revision}`; deterministic and tenant-scoped. |
|
||||
| `tenant` | string | Required; injected by Authority middleware and asserted by schema validator. |
|
||||
| `source.vendor` | string | Provider identifier (e.g., `redhat`, `osv`, `ghsa`). |
|
||||
| `source.stream` | string | Connector stream name (`csaf`, `osv`, etc.). |
|
||||
| `source.api` | string | Absolute URI of upstream document; stored for traceability. |
|
||||
| `source.collector_version` | string | Semantic version of the collector. |
|
||||
| `upstream.upstream_id` | string | Vendor- or ecosystem-provided identifier (CVE, GHSA, vendor ID). |
|
||||
| `upstream.document_version` | string | Upstream issued timestamp or revision string. |
|
||||
| `upstream.fetched_at` / `received_at` | string | ISO 8601 UTC timestamps recorded by the collector. |
|
||||
| `upstream.content_hash` | string | `sha256:` digest of the raw payload used for idempotency. |
|
||||
| `upstream.signature` | object | Required structure storing `present`, `format`, `key_id`, `sig`; even unsigned payloads set `present: false`. |
|
||||
| `content.format` | string | Source format (`CSAF`, `OSV`, etc.). |
|
||||
| `content.spec_version` | string | Upstream spec version when known. |
|
||||
| `content.raw` | object | Full upstream payload, untouched except for transport normalisation. |
|
||||
| `identifiers` | object | Normalised identifiers (`cve`, `ghsa`, `aliases`, etc.) derived losslessly from raw content. |
|
||||
| `linkset` | object | Join hints (see section 4.3). |
|
||||
| `supersedes` | string or null | Points to previous revision of same upstream doc when content hash changes. |
|
||||
|
||||
### 4.2 `vex_raw`
|
||||
|
||||
| Field | Type | Notes |
|
||||
|-------|------|-------|
|
||||
| `_id` | string | `vex_raw:{source}:{upstream_id}:{revision}`. |
|
||||
| `tenant` | string | Required; matches advisory collection requirements. |
|
||||
| `source.*` | object | Same shape and requirements as `advisory_raw`. |
|
||||
| `upstream.*` | object | Includes `document_version`, timestamps, `content_hash`, and `signature`. |
|
||||
| `content.format` | string | Typically `CycloneDX-VEX` or `CSAF-VEX`. |
|
||||
| `content.raw` | object | Entire upstream VEX payload. |
|
||||
| `identifiers.statements` | array | Normalised statement summaries (IDs, PURLs, status, justification) to accelerate policy joins. |
|
||||
| `linkset` | object | CVEs, GHSA IDs, and PURLs referenced in the document. |
|
||||
| `supersedes` | string or null | Same convention as advisory documents. |
|
||||
|
||||
### 4.3 Linkset Fields
|
||||
|
||||
- `purls`: fully qualified Package URLs extracted from raw ranges or product nodes.
|
||||
- `cpes`: Common Platform Enumerations when upstream docs provide them.
|
||||
- `aliases`: Any alternate advisory identifiers present in the payload.
|
||||
- `references`: Array of `{ type, url }` pairs pointing back to vendor advisories, patches, or exploits.
|
||||
- `reconciled_from`: Provenance of linkset entries (JSON Pointer or field origin) to make automated checks auditable.
|
||||
|
||||
Canonicalisation rules:
|
||||
- Package URLs are rendered in canonical form without qualifiers/subpaths (`pkg:type/namespace/name@version`).
|
||||
- CPE values are normalised to the 2.3 binding (`cpe:2.3:part:vendor:product:version:*:*:*:*:*:*:*`).
|
||||
|
||||
### 4.4 `advisory_observations`
|
||||
|
||||
`advisory_observations` is an immutable projection of the validated raw document used by Link‑Not‑Merge overlays. Fields mirror the JSON contract surfaced by `StellaOps.Concelier.Models.Observations.AdvisoryObservation`.
|
||||
|
||||
| Field | Type | Notes |
|
||||
|-------|------|-------|
|
||||
| `_id` | string | Deterministic observation id — `{tenant}:{source.vendor}:{upstreamId}:{revision}`. |
|
||||
| `tenant` | string | Lower-case tenant identifier. |
|
||||
| `source.vendor` / `source.stream` | string | Connector identity (e.g., `vendor/redhat`, `ecosystem/osv`). |
|
||||
| `source.api` | string | Absolute URI the connector fetched from. |
|
||||
| `source.collectorVersion` | string | Optional semantic version of the connector build. |
|
||||
| `upstream.upstream_id` | string | Advisory identifier as issued by the provider (CVE, vendor ID, etc.). |
|
||||
| `upstream.document_version` | string | Upstream revision/version string. |
|
||||
| `upstream.fetchedAt` / `upstream.receivedAt` | datetime | UTC timestamps recorded by the connector. |
|
||||
| `upstream.contentHash` | string | `sha256:` digest used for idempotency. |
|
||||
| `upstream.signature` | object | `{present, format?, keyId?, signature?}` describing upstream signature material. |
|
||||
| `content.format` / `content.specVersion` | string | Raw payload format metadata (CSAF, OSV, JSON, etc.). |
|
||||
| `content.raw` | object | Full upstream document stored losslessly (Relaxed Extended JSON). |
|
||||
| `content.metadata` | object | Optional connector-specific metadata (batch ids, hints). |
|
||||
| `linkset.aliases` | array | Normalized aliases (lower-case, sorted). |
|
||||
| `linkset.purls` | array | Normalized PURLs extracted from the document. |
|
||||
| `linkset.cpes` | array | Normalized CPE URIs. |
|
||||
| `linkset.references` | array | `{ type, url }` pairs (type lower-case). |
|
||||
| `createdAt` | datetime | Timestamp when Concelier persisted the observation. |
|
||||
| `attributes` | object | Optional provenance attributes keyed by connector. |
|
||||
|
||||
## 5. Error Model
|
||||
|
||||
| Code | Description | HTTP status | Surfaces |
|
||||
|------|-------------|-------------|----------|
|
||||
| `ERR_AOC_001` | Forbidden field detected (severity, cvss, effective data). | 400 | Ingestion APIs, CLI verifier, CI guard. |
|
||||
| `ERR_AOC_002` | Merge attempt detected (multiple upstream sources fused into one document). | 400 | Ingestion APIs, CLI verifier. |
|
||||
| `ERR_AOC_003` | Idempotency violation (duplicate without supersedes pointer). | 409 | Repository guard, Mongo unique index, CLI verifier. |
|
||||
| `ERR_AOC_004` | Missing provenance metadata (`source`, `upstream`, `signature`). | 422 | Schema validator, ingestion endpoints. |
|
||||
| `ERR_AOC_005` | Signature or checksum mismatch. | 422 | Collector validation, CLI verifier. |
|
||||
| `ERR_AOC_006` | Attempt to persist derived findings from ingestion context. | 403 | Policy engine guard, Authority scopes. |
|
||||
| `ERR_AOC_007` | Unknown top-level fields (schema violation). | 400 | Mongo validator, CLI verifier. |
|
||||
|
||||
Consumers should map these codes to CLI exit codes and structured log events so automation can fail fast and produce actionable guidance.
|
||||
|
||||
## 6. API and Tooling Interfaces
|
||||
|
||||
- **Concelier ingestion** (`StellaOps.Concelier.WebService`)
|
||||
- `POST /ingest/advisory`: accepts upstream payload metadata; server-side guard constructs and persists raw document.
|
||||
- `GET /advisories/raw/{id}` and filterable list endpoints expose raw documents for debugging and offline analysis.
|
||||
- `POST /aoc/verify`: runs guard checks over recent documents and returns summary totals plus first violations.
|
||||
- **Excititor ingestion** (`StellaOps.Excititor.WebService`) mirrors the same surface for VEX documents.
|
||||
- **CLI workflows** (`stella aoc verify`, `stella sources ingest --dry-run`) surface pre-flight verification; documentation will live in `/docs/cli/` alongside Sprint 19 CLI updates.
|
||||
- **Authority scopes**: new `advisory:ingest`, `advisory:read`, `vex:ingest`, and `vex:read` scopes enforce least privilege; see [Authority Architecture](../ARCHITECTURE_AUTHORITY.md) for scope grammar.
|
||||
|
||||
## 7. Idempotency and Supersedes Rules
|
||||
|
||||
1. Compute `content_hash` before any transformation; use it with `(source.vendor, upstream.upstream_id)` to detect duplicates.
|
||||
2. If a document with the same hash already exists, skip the write and log a no-op.
|
||||
3. When a new hash arrives for an existing upstream document, insert a new record and set `supersedes` to the previous `_id`.
|
||||
4. Keep supersedes chains acyclic; collectors must resolve conflicts by rewinding before they insert.
|
||||
5. Expose idempotency counters via metrics (`ingestion_write_total{result=ok|noop}`) to catch regressions early.
|
||||
|
||||
## 8. Migration Playbook
|
||||
|
||||
1. Freeze ingestion writes except for raw pass-through paths while deploying schema validators.
|
||||
2. Snapshot existing collections to `_backup_*` for rollback safety.
|
||||
3. Strip forbidden fields from historical documents into a temporary `advisory_view_legacy` used only during transition.
|
||||
4. Enable Mongo JSON schema validators for `advisory_raw` and `vex_raw`.
|
||||
5. Run collectors in `--dry-run` to confirm only allowed keys appear; fix violations before lifting the freeze.
|
||||
6. Point Policy Engine to consume exclusively from raw collections and compute derived outputs downstream.
|
||||
7. Delete legacy normalisation paths from ingestion code and enable runtime guards plus CI linting.
|
||||
8. Roll forward CLI, Console, and dashboards so operators can monitor AOC status end-to-end.
|
||||
|
||||
## 9. Observability and Diagnostics
|
||||
|
||||
- **Metrics**: `ingestion_write_total{result=ok|reject}`, `aoc_violation_total{code}`, `ingestion_signature_verified_total{result}`, `ingestion_latency_seconds`, `advisory_revision_count`.
|
||||
- **Traces**: spans `ingest.fetch`, `ingest.transform`, `ingest.write`, and `aoc.guard` with correlation IDs shared across workers.
|
||||
- **Logs**: structured entries must include `tenant`, `source.vendor`, `upstream.upstream_id`, `content_hash`, and `violation_code` when applicable.
|
||||
- **Dashboards**: DevOps should add panels for violation counts, signature failures, supersedes growth, and CLI verifier outcomes for each tenant.
|
||||
|
||||
## 10. Security and Tenancy Checklist
|
||||
|
||||
- Enforce Authority scopes (`advisory:ingest`, `vex:ingest`, `advisory:read`, `vex:read`) and require tenant claims on every request.
|
||||
- Maintain pinned trust stores for signature verification; capture verification result in metrics and logs.
|
||||
- Ensure collectors never log secrets or raw authentication headers; redact tokens before persistence.
|
||||
- Validate that Policy Engine remains the only identity with permission to write `effective_finding_*` documents.
|
||||
- Verify offline bundles include the raw collections, guard configuration, and verifier binaries so air-gapped installs can audit parity.
|
||||
- Document operator steps for recovering from violations, including rollback to superseded revisions and re-running policy evaluation.
|
||||
|
||||
## 11. Compliance Checklist
|
||||
|
||||
- [ ] Deterministic guard enabled in Concelier and Excititor repositories.
|
||||
- [ ] Mongo validators deployed for `advisory_raw` and `vex_raw`.
|
||||
- [ ] Authority scopes and tenant enforcement verified via integration tests.
|
||||
- [ ] CLI and CI pipelines run `stella aoc verify` against seeded snapshots.
|
||||
- [ ] Observability feeds (metrics, logs, traces) wired into dashboards with alerts.
|
||||
- [ ] Offline kit instructions updated to bundle validators and verifier tooling.
|
||||
- [ ] Security review recorded covering ingestion, tenancy, and rollback procedures.
|
||||
|
||||
---
|
||||
|
||||
*Last updated: 2025-10-27 (Sprint 19).*
|
||||
# Aggregation-Only Contract Reference
|
||||
|
||||
> The Aggregation-Only Contract (AOC) is the governing rule set that keeps StellaOps ingestion services deterministic, policy-neutral, and auditable. It applies to Concelier, Excititor, and any future collectors that write raw advisory or VEX documents.
|
||||
|
||||
## 1. Purpose and Scope
|
||||
|
||||
- Defines the canonical behaviour for `advisory_raw` and `vex_raw` collections and the linkset hints they may emit.
|
||||
- Applies to every ingestion runtime (`StellaOps.Concelier.*`, `StellaOps.Excititor.*`), the Authority scopes that guard them, and the DevOps/QA surfaces that verify compliance.
|
||||
- Complements the high-level architecture in [Concelier](../ARCHITECTURE_CONCELIER.md) and Authority enforcement documented in [Authority Architecture](../ARCHITECTURE_AUTHORITY.md).
|
||||
- Paired guidance: see the guard-rail checkpoints in [AOC Guardrails](../aoc/aoc-guardrails.md) and CLI usage that will land in `/docs/cli/` as part of Sprint 19 follow-up.
|
||||
|
||||
## 2. Philosophy and Goals
|
||||
|
||||
- Preserve upstream truth: ingestion only captures immutable raw facts plus provenance, never derived severity or policy decisions.
|
||||
- Defer interpretation: Policy Engine and downstream overlays remain the sole writers of materialised findings, severity, consensus, or risk scores.
|
||||
- Make every write explainable: provenance, signatures, and content hashes are required so operators can prove where each fact originated.
|
||||
- Keep outputs reproducible: identical inputs must yield identical documents, hashes, and linksets across replays and air-gapped installs.
|
||||
|
||||
## 3. Contract Invariants
|
||||
|
||||
| # | Invariant | What it forbids or requires | Enforcement surfaces |
|
||||
|---|-----------|-----------------------------|----------------------|
|
||||
| 1 | No derived severity at ingest | Reject top-level keys such as `severity`, `cvss`, `effective_status`, `consensus_provider`, `risk_score`. Raw upstream CVSS remains inside `content.raw`. | Mongo schema validator, `AOCWriteGuard`, Roslyn analyzer, `stella aoc verify`. |
|
||||
| 2 | No merges or opinionated dedupe | Each upstream document persists on its own; ingestion never collapses multiple vendors into one document. | Repository interceptors, unit/fixture suites. |
|
||||
| 3 | Provenance is mandatory | `source.*`, `upstream.*`, and `signature` metadata must be present; missing provenance triggers `ERR_AOC_004`. | Schema validator, guard, CLI verifier. |
|
||||
| 4 | Idempotent upserts | Writes keyed by `(vendor, upstream_id, content_hash)` either no-op or insert a new revision with `supersedes`. Duplicate hashes map to the same document. | Repository guard, storage unique index, CI smoke tests. |
|
||||
| 5 | Append-only revisions | Updates create a new document with `supersedes` pointer; no in-place mutation of content. | Mongo schema (`supersedes` format), guard, data migration scripts. |
|
||||
| 6 | Linkset only | Ingestion may compute link hints (`purls`, `cpes`, IDs) to accelerate joins, but must not transform or infer severity or policy. | Linkset builders reviewed via fixtures and analyzers. |
|
||||
| 7 | Policy-only effective findings | Only Policy Engine identities can write `effective_finding_*`; ingestion callers receive `ERR_AOC_006` if they attempt it. | Authority scopes, Policy Engine guard. |
|
||||
| 8 | Schema safety | Unknown top-level keys reject with `ERR_AOC_007`; timestamps use ISO 8601 UTC strings; tenant is required. | Mongo validator, JSON schema tests. |
|
||||
| 9 | Clock discipline | Collectors stamp `fetched_at` and `received_at` monotonically per batch to support reproducibility windows. | Collector contracts, QA fixtures. |
|
||||
|
||||
## 4. Raw Schemas
|
||||
|
||||
### 4.1 `advisory_raw`
|
||||
|
||||
| Field | Type | Notes |
|
||||
|-------|------|-------|
|
||||
| `_id` | string | `advisory_raw:{source}:{upstream_id}:{revision}`; deterministic and tenant-scoped. |
|
||||
| `tenant` | string | Required; injected by Authority middleware and asserted by schema validator. |
|
||||
| `source.vendor` | string | Provider identifier (e.g., `redhat`, `osv`, `ghsa`). |
|
||||
| `source.stream` | string | Connector stream name (`csaf`, `osv`, etc.). |
|
||||
| `source.api` | string | Absolute URI of upstream document; stored for traceability. |
|
||||
| `source.collector_version` | string | Semantic version of the collector. |
|
||||
| `upstream.upstream_id` | string | Vendor- or ecosystem-provided identifier (CVE, GHSA, vendor ID). |
|
||||
| `upstream.document_version` | string | Upstream issued timestamp or revision string. |
|
||||
| `upstream.fetched_at` / `received_at` | string | ISO 8601 UTC timestamps recorded by the collector. |
|
||||
| `upstream.content_hash` | string | `sha256:` digest of the raw payload used for idempotency. |
|
||||
| `upstream.signature` | object | Required structure storing `present`, `format`, `key_id`, `sig`; even unsigned payloads set `present: false`. |
|
||||
| `content.format` | string | Source format (`CSAF`, `OSV`, etc.). |
|
||||
| `content.spec_version` | string | Upstream spec version when known. |
|
||||
| `content.raw` | object | Full upstream payload, untouched except for transport normalisation. |
|
||||
| `identifiers` | object | Normalised identifiers (`cve`, `ghsa`, `aliases`, etc.) derived losslessly from raw content. |
|
||||
| `linkset` | object | Join hints (see section 4.3). |
|
||||
| `supersedes` | string or null | Points to previous revision of same upstream doc when content hash changes. |
|
||||
|
||||
### 4.2 `vex_raw`
|
||||
|
||||
| Field | Type | Notes |
|
||||
|-------|------|-------|
|
||||
| `_id` | string | `vex_raw:{source}:{upstream_id}:{revision}`. |
|
||||
| `tenant` | string | Required; matches advisory collection requirements. |
|
||||
| `source.*` | object | Same shape and requirements as `advisory_raw`. |
|
||||
| `upstream.*` | object | Includes `document_version`, timestamps, `content_hash`, and `signature`. |
|
||||
| `content.format` | string | Typically `CycloneDX-VEX` or `CSAF-VEX`. |
|
||||
| `content.raw` | object | Entire upstream VEX payload. |
|
||||
| `identifiers.statements` | array | Normalised statement summaries (IDs, PURLs, status, justification) to accelerate policy joins. |
|
||||
| `linkset` | object | CVEs, GHSA IDs, and PURLs referenced in the document. |
|
||||
| `supersedes` | string or null | Same convention as advisory documents. |
|
||||
|
||||
### 4.3 Linkset Fields
|
||||
|
||||
- `purls`: fully qualified Package URLs extracted from raw ranges or product nodes.
|
||||
- `cpes`: Common Platform Enumerations when upstream docs provide them.
|
||||
- `aliases`: Any alternate advisory identifiers present in the payload.
|
||||
- `references`: Array of `{ type, url }` pairs pointing back to vendor advisories, patches, or exploits.
|
||||
- `reconciled_from`: Provenance of linkset entries (JSON Pointer or field origin) to make automated checks auditable.
|
||||
|
||||
Canonicalisation rules:
|
||||
- Package URLs are rendered in canonical form without qualifiers/subpaths (`pkg:type/namespace/name@version`).
|
||||
- CPE values are normalised to the 2.3 binding (`cpe:2.3:part:vendor:product:version:*:*:*:*:*:*:*`).
|
||||
|
||||
### 4.4 `advisory_observations`
|
||||
|
||||
`advisory_observations` is an immutable projection of the validated raw document used by Link‑Not‑Merge overlays. Fields mirror the JSON contract surfaced by `StellaOps.Concelier.Models.Observations.AdvisoryObservation`.
|
||||
|
||||
| Field | Type | Notes |
|
||||
|-------|------|-------|
|
||||
| `_id` | string | Deterministic observation id — `{tenant}:{source.vendor}:{upstreamId}:{revision}`. |
|
||||
| `tenant` | string | Lower-case tenant identifier. |
|
||||
| `source.vendor` / `source.stream` | string | Connector identity (e.g., `vendor/redhat`, `ecosystem/osv`). |
|
||||
| `source.api` | string | Absolute URI the connector fetched from. |
|
||||
| `source.collectorVersion` | string | Optional semantic version of the connector build. |
|
||||
| `upstream.upstream_id` | string | Advisory identifier as issued by the provider (CVE, vendor ID, etc.). |
|
||||
| `upstream.document_version` | string | Upstream revision/version string. |
|
||||
| `upstream.fetchedAt` / `upstream.receivedAt` | datetime | UTC timestamps recorded by the connector. |
|
||||
| `upstream.contentHash` | string | `sha256:` digest used for idempotency. |
|
||||
| `upstream.signature` | object | `{present, format?, keyId?, signature?}` describing upstream signature material. |
|
||||
| `content.format` / `content.specVersion` | string | Raw payload format metadata (CSAF, OSV, JSON, etc.). |
|
||||
| `content.raw` | object | Full upstream document stored losslessly (Relaxed Extended JSON). |
|
||||
| `content.metadata` | object | Optional connector-specific metadata (batch ids, hints). |
|
||||
| `linkset.aliases` | array | Normalized aliases (lower-case, sorted). |
|
||||
| `linkset.purls` | array | Normalized PURLs extracted from the document. |
|
||||
| `linkset.cpes` | array | Normalized CPE URIs. |
|
||||
| `linkset.references` | array | `{ type, url }` pairs (type lower-case). |
|
||||
| `createdAt` | datetime | Timestamp when Concelier persisted the observation. |
|
||||
| `attributes` | object | Optional provenance attributes keyed by connector. |
|
||||
|
||||
## 5. Error Model
|
||||
|
||||
| Code | Description | HTTP status | Surfaces |
|
||||
|------|-------------|-------------|----------|
|
||||
| `ERR_AOC_001` | Forbidden field detected (severity, cvss, effective data). | 400 | Ingestion APIs, CLI verifier, CI guard. |
|
||||
| `ERR_AOC_002` | Merge attempt detected (multiple upstream sources fused into one document). | 400 | Ingestion APIs, CLI verifier. |
|
||||
| `ERR_AOC_003` | Idempotency violation (duplicate without supersedes pointer). | 409 | Repository guard, Mongo unique index, CLI verifier. |
|
||||
| `ERR_AOC_004` | Missing provenance metadata (`source`, `upstream`, `signature`). | 422 | Schema validator, ingestion endpoints. |
|
||||
| `ERR_AOC_005` | Signature or checksum mismatch. | 422 | Collector validation, CLI verifier. |
|
||||
| `ERR_AOC_006` | Attempt to persist derived findings from ingestion context. | 403 | Policy engine guard, Authority scopes. |
|
||||
| `ERR_AOC_007` | Unknown top-level fields (schema violation). | 400 | Mongo validator, CLI verifier. |
|
||||
|
||||
Consumers should map these codes to CLI exit codes and structured log events so automation can fail fast and produce actionable guidance.
|
||||
|
||||
## 6. API and Tooling Interfaces
|
||||
|
||||
- **Concelier ingestion** (`StellaOps.Concelier.WebService`)
|
||||
- `POST /ingest/advisory`: accepts upstream payload metadata; server-side guard constructs and persists raw document.
|
||||
- `GET /advisories/raw/{id}` and filterable list endpoints expose raw documents for debugging and offline analysis.
|
||||
- `POST /aoc/verify`: runs guard checks over recent documents and returns summary totals plus first violations.
|
||||
- **Excititor ingestion** (`StellaOps.Excititor.WebService`) mirrors the same surface for VEX documents.
|
||||
- **CLI workflows** (`stella aoc verify`, `stella sources ingest --dry-run`) surface pre-flight verification; documentation will live in `/docs/cli/` alongside Sprint 19 CLI updates.
|
||||
- **Authority scopes**: new `advisory:ingest`, `advisory:read`, `vex:ingest`, and `vex:read` scopes enforce least privilege; see [Authority Architecture](../ARCHITECTURE_AUTHORITY.md) for scope grammar.
|
||||
|
||||
## 7. Idempotency and Supersedes Rules
|
||||
|
||||
1. Compute `content_hash` before any transformation; use it with `(source.vendor, upstream.upstream_id)` to detect duplicates.
|
||||
2. If a document with the same hash already exists, skip the write and log a no-op.
|
||||
3. When a new hash arrives for an existing upstream document, insert a new record and set `supersedes` to the previous `_id`.
|
||||
4. Keep supersedes chains acyclic; collectors must resolve conflicts by rewinding before they insert.
|
||||
5. Expose idempotency counters via metrics (`ingestion_write_total{result=ok|noop}`) to catch regressions early.
|
||||
|
||||
## 8. Migration Playbook
|
||||
|
||||
1. Freeze ingestion writes except for raw pass-through paths while deploying schema validators.
|
||||
2. Snapshot existing collections to `_backup_*` for rollback safety.
|
||||
3. Strip forbidden fields from historical documents into a temporary `advisory_view_legacy` used only during transition.
|
||||
4. Enable Mongo JSON schema validators for `advisory_raw` and `vex_raw`.
|
||||
5. Run collectors in `--dry-run` to confirm only allowed keys appear; fix violations before lifting the freeze.
|
||||
6. Point Policy Engine to consume exclusively from raw collections and compute derived outputs downstream.
|
||||
7. Delete legacy normalisation paths from ingestion code and enable runtime guards plus CI linting.
|
||||
8. Roll forward CLI, Console, and dashboards so operators can monitor AOC status end-to-end.
|
||||
|
||||
## 9. Observability and Diagnostics
|
||||
|
||||
- **Metrics**: `ingestion_write_total{result=ok|reject}`, `aoc_violation_total{code}`, `ingestion_signature_verified_total{result}`, `ingestion_latency_seconds`, `advisory_revision_count`.
|
||||
- **Traces**: spans `ingest.fetch`, `ingest.transform`, `ingest.write`, and `aoc.guard` with correlation IDs shared across workers.
|
||||
- **Logs**: structured entries must include `tenant`, `source.vendor`, `upstream.upstream_id`, `content_hash`, and `violation_code` when applicable.
|
||||
- **Dashboards**: DevOps should add panels for violation counts, signature failures, supersedes growth, and CLI verifier outcomes for each tenant.
|
||||
|
||||
## 10. Security and Tenancy Checklist
|
||||
|
||||
- Enforce Authority scopes (`advisory:ingest`, `vex:ingest`, `advisory:read`, `vex:read`) and require tenant claims on every request.
|
||||
- Maintain pinned trust stores for signature verification; capture verification result in metrics and logs.
|
||||
- Ensure collectors never log secrets or raw authentication headers; redact tokens before persistence.
|
||||
- Validate that Policy Engine remains the only identity with permission to write `effective_finding_*` documents.
|
||||
- Verify offline bundles include the raw collections, guard configuration, and verifier binaries so air-gapped installs can audit parity.
|
||||
- Document operator steps for recovering from violations, including rollback to superseded revisions and re-running policy evaluation.
|
||||
|
||||
## 11. Compliance Checklist
|
||||
|
||||
- [ ] Deterministic guard enabled in Concelier and Excititor repositories.
|
||||
- [ ] Mongo validators deployed for `advisory_raw` and `vex_raw`.
|
||||
- [ ] Authority scopes and tenant enforcement verified via integration tests.
|
||||
- [ ] CLI and CI pipelines run `stella aoc verify` against seeded snapshots.
|
||||
- [ ] Observability feeds (metrics, logs, traces) wired into dashboards with alerts.
|
||||
- [ ] Offline kit instructions updated to bundle validators and verifier tooling.
|
||||
- [ ] Security review recorded covering ingestion, tenancy, and rollback procedures.
|
||||
|
||||
---
|
||||
|
||||
*Last updated: 2025-10-27 (Sprint 19).*
|
||||
|
||||
@@ -1,207 +1,207 @@
|
||||
# StellaOps Console — Docker Install Recipes
|
||||
|
||||
> **Audience:** Deployment Guild, Console Guild, platform operators.
|
||||
> **Scope:** Acquire the `stellaops/web-ui` image, run it with Compose or Helm, mirror it for air‑gapped environments, and keep parity with CLI workflows.
|
||||
|
||||
This guide focuses on the new **StellaOps Console** container. Start with the general [Installation Guide](../21_INSTALL_GUIDE.md) for shared prerequisites (Docker, registry access, TLS) and use the steps below to layer in the console.
|
||||
|
||||
---
|
||||
|
||||
## 1 · Release artefacts
|
||||
|
||||
| Artefact | Source | Verification |
|
||||
|----------|--------|--------------|
|
||||
| Console image | `registry.stella-ops.org/stellaops/web-ui@sha256:<digest>` | Listed in `deploy/releases/<channel>.yaml` (`yq '.services[] | select(.name=="web-ui") | .image'`). Signed with Cosign (`cosign verify --key https://stella-ops.org/keys/cosign.pub …`). |
|
||||
| Compose bundles | `deploy/compose/docker-compose.{dev,stage,prod,airgap}.yaml` | Each profile already includes a `web-ui` service pinned to the release digest. Run `docker compose --env-file <env> -f docker-compose.<profile>.yaml config` to confirm the digest matches the manifest. |
|
||||
| Helm values | `deploy/helm/stellaops/values-*.yaml` (`services.web-ui`) | CI lints the chart; use `helm template` to confirm the rendered Deployment/Service carry the expected digest and env vars. |
|
||||
| Offline artefact (preview) | Generated via `oras copy registry.stella-ops.org/stellaops/web-ui@sha256:<digest> oci-archive:stellaops-web-ui-<channel>.tar` | Record SHA-256 in the downloads manifest (`DOWNLOADS-CONSOLE-23-001`) and sign with Cosign before shipping in the Offline Kit. |
|
||||
|
||||
> **Tip:** Keep Compose/Helm digests in sync with the release manifest to preserve determinism. `deploy/tools/validate-profiles.sh` performs a quick cross-check.
|
||||
|
||||
---
|
||||
|
||||
## 2 · Compose quickstart (connected host)
|
||||
|
||||
1. **Prepare workspace**
|
||||
|
||||
```bash
|
||||
mkdir stella-console && cd stella-console
|
||||
cp /path/to/repo/deploy/compose/env/dev.env.example .env
|
||||
```
|
||||
|
||||
2. **Add console configuration** – append the following to `.env` (adjust per environment):
|
||||
|
||||
```bash
|
||||
CONSOLE_PUBLIC_BASE_URL=https://console.dev.stella-ops.local
|
||||
CONSOLE_GATEWAY_BASE_URL=https://api.dev.stella-ops.local
|
||||
AUTHORITY_ISSUER=https://authority.dev.stella-ops.local
|
||||
AUTHORITY_CLIENT_ID=console-ui
|
||||
AUTHORITY_SCOPES="ui.read ui.admin findings:read advisory:read vex:read aoc:verify"
|
||||
AUTHORITY_DPOP_ENABLED=true
|
||||
```
|
||||
|
||||
Optional extras from [`docs/deploy/console.md`](../deploy/console.md):
|
||||
|
||||
```bash
|
||||
CONSOLE_FEATURE_FLAGS=runs,downloads,policies
|
||||
CONSOLE_METRICS_ENABLED=true
|
||||
CONSOLE_LOG_LEVEL=Information
|
||||
```
|
||||
|
||||
3. **Verify bundle provenance**
|
||||
|
||||
```bash
|
||||
cosign verify-blob \
|
||||
--key https://stella-ops.org/keys/cosign.pub \
|
||||
--signature /path/to/repo/deploy/compose/docker-compose.dev.yaml.sig \
|
||||
/path/to/repo/deploy/compose/docker-compose.dev.yaml
|
||||
```
|
||||
|
||||
4. **Launch infrastructure + console**
|
||||
|
||||
```bash
|
||||
docker compose --env-file .env -f /path/to/repo/deploy/compose/docker-compose.dev.yaml up -d mongo minio
|
||||
docker compose --env-file .env -f /path/to/repo/deploy/compose/docker-compose.dev.yaml up -d web-ui
|
||||
```
|
||||
|
||||
The `web-ui` service exposes the console on port `8443` by default. Change the published port in the Compose file if you need to front it with an existing reverse proxy.
|
||||
|
||||
5. **Health check**
|
||||
|
||||
```bash
|
||||
curl -k https://console.dev.stella-ops.local/health/ready
|
||||
```
|
||||
|
||||
Expect `{"status":"Ready"}`. If the response is `401`, confirm Authority credentials and scopes.
|
||||
|
||||
---
|
||||
|
||||
## 3 · Helm deployment (cluster)
|
||||
|
||||
1. **Create an overlay** (example `console-values.yaml`):
|
||||
|
||||
```yaml
|
||||
global:
|
||||
release:
|
||||
version: "2025.10.0-edge"
|
||||
services:
|
||||
web-ui:
|
||||
image: registry.stella-ops.org/stellaops/web-ui@sha256:38b225fa7767a5b94ebae4dae8696044126aac429415e93de514d5dd95748dcf
|
||||
service:
|
||||
port: 8443
|
||||
env:
|
||||
CONSOLE_PUBLIC_BASE_URL: "https://console.dev.stella-ops.local"
|
||||
CONSOLE_GATEWAY_BASE_URL: "https://api.dev.stella-ops.local"
|
||||
AUTHORITY_ISSUER: "https://authority.dev.stella-ops.local"
|
||||
AUTHORITY_CLIENT_ID: "console-ui"
|
||||
AUTHORITY_SCOPES: "ui.read ui.admin findings:read advisory:read vex:read aoc:verify"
|
||||
AUTHORITY_DPOP_ENABLED: "true"
|
||||
CONSOLE_FEATURE_FLAGS: "runs,downloads,policies"
|
||||
CONSOLE_METRICS_ENABLED: "true"
|
||||
```
|
||||
|
||||
2. **Render and validate**
|
||||
|
||||
```bash
|
||||
helm template stella-console ./deploy/helm/stellaops -f console-values.yaml | \
|
||||
grep -A2 'name: stellaops-web-ui' -A6 'image:'
|
||||
```
|
||||
|
||||
3. **Deploy**
|
||||
|
||||
```bash
|
||||
helm upgrade --install stella-console ./deploy/helm/stellaops \
|
||||
-f deploy/helm/stellaops/values-dev.yaml \
|
||||
-f console-values.yaml
|
||||
```
|
||||
|
||||
4. **Post-deploy checks**
|
||||
|
||||
```bash
|
||||
kubectl get pods -l app.kubernetes.io/name=stellaops-web-ui
|
||||
kubectl port-forward deploy/stellaops-web-ui 8443:8443
|
||||
curl -k https://localhost:8443/health/ready
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 4 · Offline packaging
|
||||
|
||||
1. **Mirror the image to an OCI archive**
|
||||
|
||||
```bash
|
||||
DIGEST=$(yq '.services[] | select(.name=="web-ui") | .image' deploy/releases/2025.10-edge.yaml | cut -d@ -f2)
|
||||
oras copy registry.stella-ops.org/stellaops/web-ui@${DIGEST} \
|
||||
oci-archive:stellaops-web-ui-2025.10.0.tar
|
||||
shasum -a 256 stellaops-web-ui-2025.10.0.tar
|
||||
```
|
||||
|
||||
2. **Sign the archive**
|
||||
|
||||
```bash
|
||||
cosign sign-blob --key ~/keys/offline-kit.cosign \
|
||||
--output-signature stellaops-web-ui-2025.10.0.tar.sig \
|
||||
stellaops-web-ui-2025.10.0.tar
|
||||
```
|
||||
|
||||
3. **Load in the air-gap**
|
||||
|
||||
```bash
|
||||
docker load --input stellaops-web-ui-2025.10.0.tar
|
||||
docker tag stellaops/web-ui@${DIGEST} registry.airgap.local/stellaops/web-ui:2025.10.0
|
||||
```
|
||||
|
||||
4. **Update the Offline Kit manifest** (once the downloads pipeline lands):
|
||||
|
||||
```bash
|
||||
jq '.artifacts.console.webUi = {
|
||||
"digest": "sha256:'"${DIGEST#sha256:}"'",
|
||||
"archive": "stellaops-web-ui-2025.10.0.tar",
|
||||
"signature": "stellaops-web-ui-2025.10.0.tar.sig"
|
||||
}' downloads/manifest.json > downloads/manifest.json.tmp
|
||||
mv downloads/manifest.json.tmp downloads/manifest.json
|
||||
```
|
||||
|
||||
Re-run `stella offline kit import downloads/manifest.json` to validate signatures inside the air‑gapped environment.
|
||||
|
||||
---
|
||||
|
||||
## 5 · CLI parity
|
||||
|
||||
Console operations map directly to scriptable workflows:
|
||||
|
||||
| Action | CLI path |
|
||||
|--------|----------|
|
||||
| Fetch signed manifest entry | `stella downloads manifest show --artifact console/web-ui` *(CLI task `CONSOLE-DOC-23-502`, pending release)* |
|
||||
| Mirror digest to OCI archive | `stella downloads mirror --artifact console/web-ui --to oci-archive:stellaops-web-ui.tar` *(planned alongside CLI AOC parity)* |
|
||||
| Import offline kit | `stella offline kit import stellaops-web-ui-2025.10.0.tar` |
|
||||
| Validate console health | `stella console status --endpoint https://console.dev.stella-ops.local` *(planned; fallback to `curl` as shown above)* |
|
||||
|
||||
Track progress for the CLI commands via `DOCS-CONSOLE-23-014` (CLI vs UI parity matrix).
|
||||
|
||||
---
|
||||
|
||||
## 6 · Compliance checklist
|
||||
|
||||
- [ ] Image digest validated against the current release manifest.
|
||||
- [ ] Compose/Helm deployments verified with `docker compose config` / `helm template`.
|
||||
- [ ] Authority issuer, scopes, and DPoP settings documented and applied.
|
||||
- [ ] Offline archive mirrored, signed, and recorded in the downloads manifest.
|
||||
- [ ] CLI parity notes linked to the upcoming `docs/cli-vs-ui-parity.md` matrix.
|
||||
- [ ] References cross-checked with `docs/deploy/console.md` and `docs/security/console-security.md`.
|
||||
- [ ] Health checks documented for connected and air-gapped installs.
|
||||
|
||||
---
|
||||
|
||||
## 7 · References
|
||||
|
||||
- `deploy/releases/<channel>.yaml` – Release manifest (digests, SBOM metadata).
|
||||
- `deploy/compose/README.md` – Compose profile overview.
|
||||
- `deploy/helm/stellaops/values-*.yaml` – Helm defaults per environment.
|
||||
- `/docs/deploy/console.md` – Detailed environment variables, CSP, health checks.
|
||||
- `/docs/security/console-security.md` – Auth flows, scopes, DPoP, monitoring.
|
||||
- `/docs/ui/downloads.md` – Downloads manifest workflow and offline parity guidance.
|
||||
|
||||
---
|
||||
|
||||
*Last updated: 2025-10-28 (Sprint 23).*
|
||||
# StellaOps Console — Docker Install Recipes
|
||||
|
||||
> **Audience:** Deployment Guild, Console Guild, platform operators.
|
||||
> **Scope:** Acquire the `stellaops/web-ui` image, run it with Compose or Helm, mirror it for air‑gapped environments, and keep parity with CLI workflows.
|
||||
|
||||
This guide focuses on the new **StellaOps Console** container. Start with the general [Installation Guide](../21_INSTALL_GUIDE.md) for shared prerequisites (Docker, registry access, TLS) and use the steps below to layer in the console.
|
||||
|
||||
---
|
||||
|
||||
## 1 · Release artefacts
|
||||
|
||||
| Artefact | Source | Verification |
|
||||
|----------|--------|--------------|
|
||||
| Console image | `registry.stella-ops.org/stellaops/web-ui@sha256:<digest>` | Listed in `deploy/releases/<channel>.yaml` (`yq '.services[] | select(.name=="web-ui") | .image'`). Signed with Cosign (`cosign verify --key https://stella-ops.org/keys/cosign.pub …`). |
|
||||
| Compose bundles | `deploy/compose/docker-compose.{dev,stage,prod,airgap}.yaml` | Each profile already includes a `web-ui` service pinned to the release digest. Run `docker compose --env-file <env> -f docker-compose.<profile>.yaml config` to confirm the digest matches the manifest. |
|
||||
| Helm values | `deploy/helm/stellaops/values-*.yaml` (`services.web-ui`) | CI lints the chart; use `helm template` to confirm the rendered Deployment/Service carry the expected digest and env vars. |
|
||||
| Offline artefact (preview) | Generated via `oras copy registry.stella-ops.org/stellaops/web-ui@sha256:<digest> oci-archive:stellaops-web-ui-<channel>.tar` | Record SHA-256 in the downloads manifest (`DOWNLOADS-CONSOLE-23-001`) and sign with Cosign before shipping in the Offline Kit. |
|
||||
|
||||
> **Tip:** Keep Compose/Helm digests in sync with the release manifest to preserve determinism. `deploy/tools/validate-profiles.sh` performs a quick cross-check.
|
||||
|
||||
---
|
||||
|
||||
## 2 · Compose quickstart (connected host)
|
||||
|
||||
1. **Prepare workspace**
|
||||
|
||||
```bash
|
||||
mkdir stella-console && cd stella-console
|
||||
cp /path/to/repo/deploy/compose/env/dev.env.example .env
|
||||
```
|
||||
|
||||
2. **Add console configuration** – append the following to `.env` (adjust per environment):
|
||||
|
||||
```bash
|
||||
CONSOLE_PUBLIC_BASE_URL=https://console.dev.stella-ops.local
|
||||
CONSOLE_GATEWAY_BASE_URL=https://api.dev.stella-ops.local
|
||||
AUTHORITY_ISSUER=https://authority.dev.stella-ops.local
|
||||
AUTHORITY_CLIENT_ID=console-ui
|
||||
AUTHORITY_SCOPES="ui.read ui.admin findings:read advisory:read vex:read aoc:verify"
|
||||
AUTHORITY_DPOP_ENABLED=true
|
||||
```
|
||||
|
||||
Optional extras from [`docs/deploy/console.md`](../deploy/console.md):
|
||||
|
||||
```bash
|
||||
CONSOLE_FEATURE_FLAGS=runs,downloads,policies
|
||||
CONSOLE_METRICS_ENABLED=true
|
||||
CONSOLE_LOG_LEVEL=Information
|
||||
```
|
||||
|
||||
3. **Verify bundle provenance**
|
||||
|
||||
```bash
|
||||
cosign verify-blob \
|
||||
--key https://stella-ops.org/keys/cosign.pub \
|
||||
--signature /path/to/repo/deploy/compose/docker-compose.dev.yaml.sig \
|
||||
/path/to/repo/deploy/compose/docker-compose.dev.yaml
|
||||
```
|
||||
|
||||
4. **Launch infrastructure + console**
|
||||
|
||||
```bash
|
||||
docker compose --env-file .env -f /path/to/repo/deploy/compose/docker-compose.dev.yaml up -d mongo minio
|
||||
docker compose --env-file .env -f /path/to/repo/deploy/compose/docker-compose.dev.yaml up -d web-ui
|
||||
```
|
||||
|
||||
The `web-ui` service exposes the console on port `8443` by default. Change the published port in the Compose file if you need to front it with an existing reverse proxy.
|
||||
|
||||
5. **Health check**
|
||||
|
||||
```bash
|
||||
curl -k https://console.dev.stella-ops.local/health/ready
|
||||
```
|
||||
|
||||
Expect `{"status":"Ready"}`. If the response is `401`, confirm Authority credentials and scopes.
|
||||
|
||||
---
|
||||
|
||||
## 3 · Helm deployment (cluster)
|
||||
|
||||
1. **Create an overlay** (example `console-values.yaml`):
|
||||
|
||||
```yaml
|
||||
global:
|
||||
release:
|
||||
version: "2025.10.0-edge"
|
||||
services:
|
||||
web-ui:
|
||||
image: registry.stella-ops.org/stellaops/web-ui@sha256:38b225fa7767a5b94ebae4dae8696044126aac429415e93de514d5dd95748dcf
|
||||
service:
|
||||
port: 8443
|
||||
env:
|
||||
CONSOLE_PUBLIC_BASE_URL: "https://console.dev.stella-ops.local"
|
||||
CONSOLE_GATEWAY_BASE_URL: "https://api.dev.stella-ops.local"
|
||||
AUTHORITY_ISSUER: "https://authority.dev.stella-ops.local"
|
||||
AUTHORITY_CLIENT_ID: "console-ui"
|
||||
AUTHORITY_SCOPES: "ui.read ui.admin findings:read advisory:read vex:read aoc:verify"
|
||||
AUTHORITY_DPOP_ENABLED: "true"
|
||||
CONSOLE_FEATURE_FLAGS: "runs,downloads,policies"
|
||||
CONSOLE_METRICS_ENABLED: "true"
|
||||
```
|
||||
|
||||
2. **Render and validate**
|
||||
|
||||
```bash
|
||||
helm template stella-console ./deploy/helm/stellaops -f console-values.yaml | \
|
||||
grep -A2 'name: stellaops-web-ui' -A6 'image:'
|
||||
```
|
||||
|
||||
3. **Deploy**
|
||||
|
||||
```bash
|
||||
helm upgrade --install stella-console ./deploy/helm/stellaops \
|
||||
-f deploy/helm/stellaops/values-dev.yaml \
|
||||
-f console-values.yaml
|
||||
```
|
||||
|
||||
4. **Post-deploy checks**
|
||||
|
||||
```bash
|
||||
kubectl get pods -l app.kubernetes.io/name=stellaops-web-ui
|
||||
kubectl port-forward deploy/stellaops-web-ui 8443:8443
|
||||
curl -k https://localhost:8443/health/ready
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 4 · Offline packaging
|
||||
|
||||
1. **Mirror the image to an OCI archive**
|
||||
|
||||
```bash
|
||||
DIGEST=$(yq '.services[] | select(.name=="web-ui") | .image' deploy/releases/2025.10-edge.yaml | cut -d@ -f2)
|
||||
oras copy registry.stella-ops.org/stellaops/web-ui@${DIGEST} \
|
||||
oci-archive:stellaops-web-ui-2025.10.0.tar
|
||||
shasum -a 256 stellaops-web-ui-2025.10.0.tar
|
||||
```
|
||||
|
||||
2. **Sign the archive**
|
||||
|
||||
```bash
|
||||
cosign sign-blob --key ~/keys/offline-kit.cosign \
|
||||
--output-signature stellaops-web-ui-2025.10.0.tar.sig \
|
||||
stellaops-web-ui-2025.10.0.tar
|
||||
```
|
||||
|
||||
3. **Load in the air-gap**
|
||||
|
||||
```bash
|
||||
docker load --input stellaops-web-ui-2025.10.0.tar
|
||||
docker tag stellaops/web-ui@${DIGEST} registry.airgap.local/stellaops/web-ui:2025.10.0
|
||||
```
|
||||
|
||||
4. **Update the Offline Kit manifest** (once the downloads pipeline lands):
|
||||
|
||||
```bash
|
||||
jq '.artifacts.console.webUi = {
|
||||
"digest": "sha256:'"${DIGEST#sha256:}"'",
|
||||
"archive": "stellaops-web-ui-2025.10.0.tar",
|
||||
"signature": "stellaops-web-ui-2025.10.0.tar.sig"
|
||||
}' downloads/manifest.json > downloads/manifest.json.tmp
|
||||
mv downloads/manifest.json.tmp downloads/manifest.json
|
||||
```
|
||||
|
||||
Re-run `stella offline kit import downloads/manifest.json` to validate signatures inside the air‑gapped environment.
|
||||
|
||||
---
|
||||
|
||||
## 5 · CLI parity
|
||||
|
||||
Console operations map directly to scriptable workflows:
|
||||
|
||||
| Action | CLI path |
|
||||
|--------|----------|
|
||||
| Fetch signed manifest entry | `stella downloads manifest show --artifact console/web-ui` *(CLI task `CONSOLE-DOC-23-502`, pending release)* |
|
||||
| Mirror digest to OCI archive | `stella downloads mirror --artifact console/web-ui --to oci-archive:stellaops-web-ui.tar` *(planned alongside CLI AOC parity)* |
|
||||
| Import offline kit | `stella offline kit import stellaops-web-ui-2025.10.0.tar` |
|
||||
| Validate console health | `stella console status --endpoint https://console.dev.stella-ops.local` *(planned; fallback to `curl` as shown above)* |
|
||||
|
||||
Track progress for the CLI commands via `DOCS-CONSOLE-23-014` (CLI vs UI parity matrix).
|
||||
|
||||
---
|
||||
|
||||
## 6 · Compliance checklist
|
||||
|
||||
- [ ] Image digest validated against the current release manifest.
|
||||
- [ ] Compose/Helm deployments verified with `docker compose config` / `helm template`.
|
||||
- [ ] Authority issuer, scopes, and DPoP settings documented and applied.
|
||||
- [ ] Offline archive mirrored, signed, and recorded in the downloads manifest.
|
||||
- [ ] CLI parity notes linked to the upcoming `docs/cli-vs-ui-parity.md` matrix.
|
||||
- [ ] References cross-checked with `docs/deploy/console.md` and `docs/security/console-security.md`.
|
||||
- [ ] Health checks documented for connected and air-gapped installs.
|
||||
|
||||
---
|
||||
|
||||
## 7 · References
|
||||
|
||||
- `deploy/releases/<channel>.yaml` – Release manifest (digests, SBOM metadata).
|
||||
- `deploy/compose/README.md` – Compose profile overview.
|
||||
- `deploy/helm/stellaops/values-*.yaml` – Helm defaults per environment.
|
||||
- `/docs/deploy/console.md` – Detailed environment variables, CSP, health checks.
|
||||
- `/docs/security/console-security.md` – Auth flows, scopes, DPoP, monitoring.
|
||||
- `/docs/ui/downloads.md` – Downloads manifest workflow and offline parity guidance.
|
||||
|
||||
---
|
||||
|
||||
*Last updated: 2025-10-28 (Sprint 23).*
|
||||
|
||||
@@ -1,118 +1,118 @@
|
||||
# Notifications Architecture
|
||||
|
||||
> **Imposed rule:** Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.
|
||||
|
||||
This dossier distils the Notify architecture into implementation-ready guidance for service owners, SREs, and integrators. It complements the high-level overview by detailing process boundaries, persistence models, and extensibility points.
|
||||
|
||||
---
|
||||
|
||||
## 1. Runtime shape
|
||||
|
||||
```
|
||||
┌──────────────────┐
|
||||
│ Authority (OpTok)│
|
||||
└───────┬──────────┘
|
||||
│
|
||||
┌───────▼──────────┐ ┌───────────────┐
|
||||
│ Notify.WebService│◀──────▶│ MongoDB │
|
||||
Tenant API│ REST + gRPC WIP │ │ rules/channels│
|
||||
└───────▲──────────┘ │ deliveries │
|
||||
│ │ digests │
|
||||
Internal bus │ └───────────────┘
|
||||
(NATS/Redis/etc) │
|
||||
│
|
||||
┌─────────▼─────────┐ ┌───────────────┐
|
||||
│ Notify.Worker │◀────▶│ Redis / Cache │
|
||||
│ rule eval + render│ │ throttles/locks│
|
||||
└─────────▲─────────┘ └───────▲───────┘
|
||||
│ │
|
||||
│ │
|
||||
┌──────┴──────┐ ┌─────────┴────────┐
|
||||
│ Connectors │──────▶│ Slack/Teams/... │
|
||||
│ (plug-ins) │ │ External targets │
|
||||
└─────────────┘ └──────────────────┘
|
||||
```
|
||||
|
||||
- **WebService** hosts REST endpoints (`/channels`, `/rules`, `/templates`, `/deliveries`, `/digests`, `/stats`) and handles schema normalisation, validation, and Authority enforcement.
|
||||
- **Worker** subscribes to the platform event bus, evaluates rules per tenant, applies throttles/digests, renders payloads, writes ledger entries, and invokes connectors.
|
||||
- **Plug-ins** live under `plugins/notify/` and are loaded deterministically at service start (`orderedPlugins` list). Each implements connector contracts and optional health/test-preview providers.
|
||||
|
||||
Both services share options via `notify.yaml` (see `etc/notify.yaml.sample`). For dev/test scenarios, an in-memory repository exists but production requires Mongo + Redis/NATS for durability and coordination.
|
||||
|
||||
---
|
||||
|
||||
## 2. Event ingestion and rule evaluation
|
||||
|
||||
1. **Subscription.** Workers attach to the internal bus (Redis Streams or NATS JetStream). Each partition key is `tenantId|scope.digest|event.kind` to preserve order for a given artefact.
|
||||
2. **Normalisation.** Incoming events are hydrated into `NotifyEvent` envelopes. Payload JSON is normalised (sorted object keys) to preserve determinism and enable hashing.
|
||||
3. **Rule snapshot.** Per-tenant rule sets are cached in memory. Change streams from Mongo trigger snapshot refreshes without restart.
|
||||
4. **Match pipeline.**
|
||||
- Tenant check (`rule.tenantId` vs. event tenant).
|
||||
- Kind/namespace/repository/digest filters.
|
||||
- Severity and KEV gating based on event deltas.
|
||||
- VEX gating using `NotifyRuleMatchVex`.
|
||||
- Action iteration with throttle/digest decisions.
|
||||
5. **Idempotency.** Each action computes `hash(ruleId|actionId|event.kind|scope.digest|delta.hash|dayBucket)`; matches within throttle TTL record `status=Throttled` and stop.
|
||||
6. **Dispatch.** If digest is `instant`, the renderer immediately processes the action. Otherwise the event is appended to the digest window for later flush.
|
||||
|
||||
Failures during evaluation are logged with correlation IDs and surfaced through `/stats` and worker metrics (`notify_rule_eval_failures_total`, `notify_digest_flush_errors_total`).
|
||||
|
||||
---
|
||||
|
||||
## 3. Rendering & connectors
|
||||
|
||||
- **Template resolution.** The renderer picks the template in this order: action template → channel default template → locale fallback → built-in minimal template. Locale negotiation reduces `en-US` to `en-us`.
|
||||
- **Helpers & partials.** Exposed helpers mirror the list in [`notifications/templates.md`](templates.md#3-variables-helpers-and-context). Plug-ins may register additional helpers but must remain deterministic and side-effect free.
|
||||
- **Rendering output.** `NotifyDeliveryRendered` captures:
|
||||
- `channelType`, `format`, `locale`
|
||||
- `title`, `body`, optional `summary`, `textBody`
|
||||
- `target` (redacted where necessary)
|
||||
- `attachments[]` (safe URLs or references)
|
||||
- `bodyHash` (lowercase SHA-256) for audit parity
|
||||
- **Connector contract.** Connectors implement `INotifyConnector` (send + health) and can implement `INotifyChannelTestProvider` for `/channels/{id}/test`. All plugs are single-tenant aware; secrets are pulled via references at send time and never persisted in Mongo.
|
||||
- **Retries.** Workers track attempts with exponential jitter. On permanent failure, deliveries are marked `Failed` with `statusReason`, and optional DLQ fan-out is slated for Sprint 40.
|
||||
|
||||
---
|
||||
|
||||
## 4. Persistence model
|
||||
|
||||
| Collection | Purpose | Key fields & indexes |
|
||||
|------------|---------|----------------------|
|
||||
| `rules` | Tenant rule definitions. | `_id`, `tenantId`, `enabled`; index on `{tenantId, enabled}`. |
|
||||
| `channels` | Channel metadata + config references. | `_id`, `tenantId`, `type`; index on `{tenantId, type}`. |
|
||||
| `templates` | Locale-specific render bodies. | `_id`, `tenantId`, `channelType`, `key`; index on `{tenantId, channelType, key}`. |
|
||||
| `deliveries` | Ledger of rendered notifications. | `_id`, `tenantId`, `sentAt`; compound index on `{tenantId, sentAt:-1}` for history queries. |
|
||||
| `digests` | Open digest windows per action. | `_id` (`tenantId:actionKey:window`), `status`; index on `{tenantId, actionKey}`. |
|
||||
| `throttles` | Short-lived throttle tokens (Mongo or Redis). | Key format `idem:<hash>` with TTL aligned to throttle duration. |
|
||||
|
||||
Documents are stored using the canonical JSON serializer (`NotifyCanonicalJsonSerializer`) to preserve property ordering and casing. Schema migration helpers upgrade stored documents when new versions ship.
|
||||
|
||||
---
|
||||
|
||||
## 5. Deployment & configuration
|
||||
|
||||
- **Configuration sources.** YAML files feed typed options (`NotifyMongoOptions`, `NotifyWorkerOptions`, etc.). Environment variables can override connection strings and rate limits for production.
|
||||
- **Authority integration.** Two OAuth clients (`notify-web`, `notify-web-dev`) with scopes `notify.read` and `notify.admin` are required. Authority enforcement can be disabled for air-gapped dev use by providing `developmentSigningKey`.
|
||||
- **Plug-in management.** `plugins.baseDirectory` and `orderedPlugins` guarantee deterministic loading. Offline Kits copy the plug-in tree verbatim; operations must keep the order aligned across environments.
|
||||
- **Observability.** Workers expose structured logs (`ruleId`, `actionId`, `eventId`, `throttleKey`). Metrics include:
|
||||
- `notify_rule_matches_total{tenant,eventKind}`
|
||||
- `notify_delivery_attempts_total{channelType,status}`
|
||||
- `notify_digest_open_windows{window}`
|
||||
- Optional OpenTelemetry traces for rule evaluation and connector round-trips.
|
||||
- **Scaling levers.** Increase worker replicas to cope with bus throughput; adjust `worker.prefetchCount` for Redis Streams or `ackWait` for NATS JetStream. WebService remains stateless and scales horizontally behind the gateway.
|
||||
|
||||
---
|
||||
|
||||
## 6. Roadmap alignment
|
||||
|
||||
| Backlog | Architectural note |
|
||||
|---------|--------------------|
|
||||
| `NOTIFY-SVC-38-001` | Standardise event envelope publication (idempotency keys) – ensure bus bindings use the documented key format. |
|
||||
| `NOTIFY-SVC-38-002..004` | Introduce simulation endpoints and throttle dashboards – expect additional `/internal/notify/simulate` routes and metrics; update once merged. |
|
||||
| `NOTIFY-SVC-39-001..004` | Correlation engine, digests generator, simulation API, quiet hours – anticipate new Mongo documents (`quietHours`, correlation caches) and connector metadata (quiet mode hints). Review this guide when implementations land. |
|
||||
|
||||
Action: schedule a documentation sync with the Notifications Service Guild immediately after `NOTIFY-SVC-39-001..004` merge to confirm schema adjustments (e.g., correlation edge storage, quiet hour calendars) and add any new persistence or API details here.
|
||||
|
||||
---
|
||||
|
||||
> **Imposed rule reminder:** Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.
|
||||
# Notifications Architecture
|
||||
|
||||
> **Imposed rule:** Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.
|
||||
|
||||
This dossier distils the Notify architecture into implementation-ready guidance for service owners, SREs, and integrators. It complements the high-level overview by detailing process boundaries, persistence models, and extensibility points.
|
||||
|
||||
---
|
||||
|
||||
## 1. Runtime shape
|
||||
|
||||
```
|
||||
┌──────────────────┐
|
||||
│ Authority (OpTok)│
|
||||
└───────┬──────────┘
|
||||
│
|
||||
┌───────▼──────────┐ ┌───────────────┐
|
||||
│ Notify.WebService│◀──────▶│ MongoDB │
|
||||
Tenant API│ REST + gRPC WIP │ │ rules/channels│
|
||||
└───────▲──────────┘ │ deliveries │
|
||||
│ │ digests │
|
||||
Internal bus │ └───────────────┘
|
||||
(NATS/Redis/etc) │
|
||||
│
|
||||
┌─────────▼─────────┐ ┌───────────────┐
|
||||
│ Notify.Worker │◀────▶│ Redis / Cache │
|
||||
│ rule eval + render│ │ throttles/locks│
|
||||
└─────────▲─────────┘ └───────▲───────┘
|
||||
│ │
|
||||
│ │
|
||||
┌──────┴──────┐ ┌─────────┴────────┐
|
||||
│ Connectors │──────▶│ Slack/Teams/... │
|
||||
│ (plug-ins) │ │ External targets │
|
||||
└─────────────┘ └──────────────────┘
|
||||
```
|
||||
|
||||
- **WebService** hosts REST endpoints (`/channels`, `/rules`, `/templates`, `/deliveries`, `/digests`, `/stats`) and handles schema normalisation, validation, and Authority enforcement.
|
||||
- **Worker** subscribes to the platform event bus, evaluates rules per tenant, applies throttles/digests, renders payloads, writes ledger entries, and invokes connectors.
|
||||
- **Plug-ins** live under `plugins/notify/` and are loaded deterministically at service start (`orderedPlugins` list). Each implements connector contracts and optional health/test-preview providers.
|
||||
|
||||
Both services share options via `notify.yaml` (see `etc/notify.yaml.sample`). For dev/test scenarios, an in-memory repository exists but production requires Mongo + Redis/NATS for durability and coordination.
|
||||
|
||||
---
|
||||
|
||||
## 2. Event ingestion and rule evaluation
|
||||
|
||||
1. **Subscription.** Workers attach to the internal bus (Redis Streams or NATS JetStream). Each partition key is `tenantId|scope.digest|event.kind` to preserve order for a given artefact.
|
||||
2. **Normalisation.** Incoming events are hydrated into `NotifyEvent` envelopes. Payload JSON is normalised (sorted object keys) to preserve determinism and enable hashing.
|
||||
3. **Rule snapshot.** Per-tenant rule sets are cached in memory. Change streams from Mongo trigger snapshot refreshes without restart.
|
||||
4. **Match pipeline.**
|
||||
- Tenant check (`rule.tenantId` vs. event tenant).
|
||||
- Kind/namespace/repository/digest filters.
|
||||
- Severity and KEV gating based on event deltas.
|
||||
- VEX gating using `NotifyRuleMatchVex`.
|
||||
- Action iteration with throttle/digest decisions.
|
||||
5. **Idempotency.** Each action computes `hash(ruleId|actionId|event.kind|scope.digest|delta.hash|dayBucket)`; matches within throttle TTL record `status=Throttled` and stop.
|
||||
6. **Dispatch.** If digest is `instant`, the renderer immediately processes the action. Otherwise the event is appended to the digest window for later flush.
|
||||
|
||||
Failures during evaluation are logged with correlation IDs and surfaced through `/stats` and worker metrics (`notify_rule_eval_failures_total`, `notify_digest_flush_errors_total`).
|
||||
|
||||
---
|
||||
|
||||
## 3. Rendering & connectors
|
||||
|
||||
- **Template resolution.** The renderer picks the template in this order: action template → channel default template → locale fallback → built-in minimal template. Locale negotiation reduces `en-US` to `en-us`.
|
||||
- **Helpers & partials.** Exposed helpers mirror the list in [`notifications/templates.md`](templates.md#3-variables-helpers-and-context). Plug-ins may register additional helpers but must remain deterministic and side-effect free.
|
||||
- **Rendering output.** `NotifyDeliveryRendered` captures:
|
||||
- `channelType`, `format`, `locale`
|
||||
- `title`, `body`, optional `summary`, `textBody`
|
||||
- `target` (redacted where necessary)
|
||||
- `attachments[]` (safe URLs or references)
|
||||
- `bodyHash` (lowercase SHA-256) for audit parity
|
||||
- **Connector contract.** Connectors implement `INotifyConnector` (send + health) and can implement `INotifyChannelTestProvider` for `/channels/{id}/test`. All plugs are single-tenant aware; secrets are pulled via references at send time and never persisted in Mongo.
|
||||
- **Retries.** Workers track attempts with exponential jitter. On permanent failure, deliveries are marked `Failed` with `statusReason`, and optional DLQ fan-out is slated for Sprint 40.
|
||||
|
||||
---
|
||||
|
||||
## 4. Persistence model
|
||||
|
||||
| Collection | Purpose | Key fields & indexes |
|
||||
|------------|---------|----------------------|
|
||||
| `rules` | Tenant rule definitions. | `_id`, `tenantId`, `enabled`; index on `{tenantId, enabled}`. |
|
||||
| `channels` | Channel metadata + config references. | `_id`, `tenantId`, `type`; index on `{tenantId, type}`. |
|
||||
| `templates` | Locale-specific render bodies. | `_id`, `tenantId`, `channelType`, `key`; index on `{tenantId, channelType, key}`. |
|
||||
| `deliveries` | Ledger of rendered notifications. | `_id`, `tenantId`, `sentAt`; compound index on `{tenantId, sentAt:-1}` for history queries. |
|
||||
| `digests` | Open digest windows per action. | `_id` (`tenantId:actionKey:window`), `status`; index on `{tenantId, actionKey}`. |
|
||||
| `throttles` | Short-lived throttle tokens (Mongo or Redis). | Key format `idem:<hash>` with TTL aligned to throttle duration. |
|
||||
|
||||
Documents are stored using the canonical JSON serializer (`NotifyCanonicalJsonSerializer`) to preserve property ordering and casing. Schema migration helpers upgrade stored documents when new versions ship.
|
||||
|
||||
---
|
||||
|
||||
## 5. Deployment & configuration
|
||||
|
||||
- **Configuration sources.** YAML files feed typed options (`NotifyMongoOptions`, `NotifyWorkerOptions`, etc.). Environment variables can override connection strings and rate limits for production.
|
||||
- **Authority integration.** Two OAuth clients (`notify-web`, `notify-web-dev`) with scopes `notify.read` and `notify.admin` are required. Authority enforcement can be disabled for air-gapped dev use by providing `developmentSigningKey`.
|
||||
- **Plug-in management.** `plugins.baseDirectory` and `orderedPlugins` guarantee deterministic loading. Offline Kits copy the plug-in tree verbatim; operations must keep the order aligned across environments.
|
||||
- **Observability.** Workers expose structured logs (`ruleId`, `actionId`, `eventId`, `throttleKey`). Metrics include:
|
||||
- `notify_rule_matches_total{tenant,eventKind}`
|
||||
- `notify_delivery_attempts_total{channelType,status}`
|
||||
- `notify_digest_open_windows{window}`
|
||||
- Optional OpenTelemetry traces for rule evaluation and connector round-trips.
|
||||
- **Scaling levers.** Increase worker replicas to cope with bus throughput; adjust `worker.prefetchCount` for Redis Streams or `ackWait` for NATS JetStream. WebService remains stateless and scales horizontally behind the gateway.
|
||||
|
||||
---
|
||||
|
||||
## 6. Roadmap alignment
|
||||
|
||||
| Backlog | Architectural note |
|
||||
|---------|--------------------|
|
||||
| `NOTIFY-SVC-38-001` | Standardise event envelope publication (idempotency keys) – ensure bus bindings use the documented key format. |
|
||||
| `NOTIFY-SVC-38-002..004` | Introduce simulation endpoints and throttle dashboards – expect additional `/internal/notify/simulate` routes and metrics; update once merged. |
|
||||
| `NOTIFY-SVC-39-001..004` | Correlation engine, digests generator, simulation API, quiet hours – anticipate new Mongo documents (`quietHours`, correlation caches) and connector metadata (quiet mode hints). Review this guide when implementations land. |
|
||||
|
||||
Action: schedule a documentation sync with the Notifications Service Guild immediately after `NOTIFY-SVC-39-001..004` merge to confirm schema adjustments (e.g., correlation edge storage, quiet hour calendars) and add any new persistence or API details here.
|
||||
|
||||
---
|
||||
|
||||
> **Imposed rule reminder:** Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user