This commit is contained in:
StellaOps Bot
2025-11-23 23:40:10 +02:00
parent c13355923f
commit 029002ad05
93 changed files with 2160 additions and 285 deletions

View File

@@ -0,0 +1,56 @@
# Concelier mirror/offline export path (dev baseline)
Goal: serve advisory chunks and provenance via the existing `/concelier/exports/mirror/*` endpoints without blocking on release signing/DevOps pipelines.
## Minimal layout (dev)
Point `CONCELIER_MIRROR__EXPORTROOT` at a directory that contains:
```
<exportId>/
mirror/
index.json
<domain>/manifest.json
<domain>/bundle.json
<domain>/bundle.json.jws (optional; unsigned in dev)
```
Example generator (dev):
```
EXPORTROOT=out/concelier/exports
EXPORTID=$(date -u +%Y%m%dT%H%M%SZ)
DOMAIN=primary
mkdir -p "$EXPORTROOT/$EXPORTID/mirror/$DOMAIN"
cat > "$EXPORTROOT/$EXPORTID/mirror/index.json" <<'JSON'
{"schemaVersion":1,"domains":[{"id":"primary","displayName":"Primary"}]}
JSON
cat > "$EXPORTROOT/$EXPORTID/mirror/$DOMAIN/manifest.json" <<'JSON'
{"domainId":"primary","created":"2025-11-23T00:00:00Z","schemaVersion":1,"advisories":0}
JSON
# Placeholder bundle built from canonical chunks; replace with real export job output
echo '{"advisories":[]}' > "$EXPORTROOT/$EXPORTID/mirror/$DOMAIN/bundle.json"
echo 'unsigned-dev-bundle' > "$EXPORTROOT/$EXPORTID/mirror/$DOMAIN/bundle.json.jws"
ln -sfn "$EXPORTID" "$EXPORTROOT/latest"
```
Configure Concelier to serve it:
```
CONCELIER_MIRROR__ENABLED=true
CONCELIER_MIRROR__EXPORTROOT=out/concelier/exports
CONCELIER_MIRROR__ACTIVEEXPORTID=<exportId> # optional; falls back to latest
CONCELIER_MIRROR__DOMAINS__0__ID=primary
CONCELIER_MIRROR__DOMAINS__0__DISPLAYNAME=Primary
CONCELIER_MIRROR__DOMAINS__0__REQUIREAUTHENTICATION=false
```
With this in place, the existing endpoints return:
- `/concelier/exports/index.json`
- `/concelier/exports/mirror/primary/manifest.json`
- `/concelier/exports/mirror/primary/bundle.json` (and `.jws`)
## Why this unblocks development
- Uses the canonical chunk schema already emitted by CONCELIER-LNM-21-001.
- Requires no release signing; works with unsigned dev bundles.
- Keeps path and filenames identical to planned release layout, so DevOps can later layer signing/TUF in a separate sprint.
## Next (DevOps) step
- `DEVOPS-MIRROR-23-001-REL` will replace the placeholder bundle generator with the signed/exported artefact pipeline and enforce DSSE/TUF.

View File

@@ -31,13 +31,13 @@ OCI=1 scripts/mirror/ci-sign.sh
```
## Temporary dev key (to unblock CI until production key is issued)
Use this throwaway Ed25519 key only for non-production runs. Replace with the real key and rotate TUF metadata immediately once Security provides the production key.
Use this throwaway Ed25519 key only for non-production runs. Generated 2025-11-23 to replace the previous placeholder; rotate TUF metadata immediately after swapping in the production key.
```
MIRROR_SIGN_KEY_B64=LS0tLS1CRUdJTiBQUklWQVRFIEtFWS0tLS0tCk1JR2pBZ0VCQkFBd0RRWUpLb1pJaHZjTkFRRUxCUUF3Z1lCakEwQmdOVkJBTVRJSEp2Y0hScGIyNHhIREFhTUE4R0ExVUVDZ3dJVkdkbFlXSnZkWEpwYm1jZ1IyOXZiR1JoYm1jd0hoY05Nakl3TlRBd05UVXpNVGMzV2hjTk1qRTFORFF3TlRVek1UYzNXakFhTVE4d0RRWURWUVFIREFKSFpuSmxaUzFwWkdWdWRDQkpiblJsYkNBeEN6QUpCZ05WQkFNTURHMWhaMkZ5WkFZRFZRUUlEQWx5YjI1MFpXNWtMV3hwYm1jZ1EwRXdIaGNOTWpBd09URTRNVEF4TmpVd1dqQllNUjh3RFFZRFZRUURFd0pIVm1WeWMybHZiaUJIYm5ScGRHVWlNQ0FHQTFVRUF3d0JiM1JoYkd4bGNpQkRiM0pwZEhrZ1EyVnlkbWxqWlhNd1doY05NakF3T1RFNE1UQXhOalV3V2pBZEJnTlZCQU1NRDhSd2IzSnNaV04wYjNKMGFXWnBZMkYwYVc5dWN6Q0NBU0l3RFFZSktvWklodmNOQVFFQkJRQURnZ0dQQURDQ0FRb0NnZ0dCUFEvQmtSUVE5aFl4MzM5L013SEdHSWVhc3Y2cEhPMHVKLy9VRE85bnpSZThndUFCeC8zRm0zYzdzODh5Z2NhSU05bmZGQkFPUHdFZm1ZeFFHUTZudUtXaVNZN0xobDlGWmxmR1FkdHRkQWJGZGFXdGFXNWV2OGxrUmZNcU92b2cyN0szdEptc2R3bUR4aHpuK0Y4WmpQbW1qa1MyT0lYUGRxZXVuSjJJQUdQUm12K0huWThRSjA2ZTBnSk1CZkZkRXhpVFpCbkdNK2hvbTBYZ24wbE1DTHpoSExsYTZIN0NQYkFqSWhZL3B4MEh2UGtaeVc2cGl0OG9acWJ5dEJBMlVwS0RGeU5OVnRvVnFZQVg0NCtaVE5EclUxWlVLajZ1ZWhtZ0p5bThZMjl2WVZyL0JUWUpBaFZNY0I4alZXSTZVUXdPQ0F3RUFBYU1tTUNRd0N3WURWUjBUQVFIL0JBVXdBd0VCL3pBTkJna3Foa2lHOXcwQkNRRVdKREFkQmdOVkhRNEVGZ1FVdUxLRjZCcXlHWmltNVBBU2ZaZXBVVEdPaEhHa3dDZ1lJS29aSXpqMEVBd0lEU0FBd1JRSWhBTCt2bmxOZkI0czYvRDdNZ3ZKblFyZlNPeDBWb1NQWUMxcU9PdHd0aXdEb3ZkRnhHSnZLY0R3WXIvQUhTMmJzRnFJMjduRzhPRERmQm4rS1ZxL1BQT3ZMTVpkTTROblVVallNWlBLMXZWQndXVGpKeXpKV3lXUmF2dnJTd2tNQmtTRmdLWW5uU1huOGFPVnhHazRyYzlzSkpEUT0KLS0tLS1FTkQgUFJJVkFURSBLRVktLS0tLQo=
MIRROR_SIGN_KEY_B64=LS0tLS1CRUdJTiBQUklWQVRFIEtFWS0tLS0tCk1DNENBUUF3QlFZREsyVndCQ0lFSURqb3pDRVdKVVFUdW1xZ2gyRmZXcVBaemlQbkdaSzRvOFZRTThGYkZCSEcKLS0tLS1FTkQgUFJJVkFURSBLRVktLS0tLQo=
```
**Do not ship with this key.** Set `REQUIRE_PROD_SIGNING=1` for release/tag builds so they fail without the real key.
**Do not ship with this key.** Set `REQUIRE_PROD_SIGNING=1` for release/tag builds so they fail without the real key. Add the production key as a Gitea secret (`MIRROR_SIGN_KEY_B64`) and rerun the workflow; remove this temporary key block once rotated.
## Verification
The CI step already runs `scripts/mirror/verify_thin_bundle.py`. For OCI, ensure `out/mirror/thin/oci/index.json` references the manifest digest.

View File

@@ -2,6 +2,7 @@
| Task ID | State | Notes |
| --- | --- | --- |
| `POLICY-ENGINE-29-002-CONTRACT` | DONE (2025-11-23) | Streaming simulation contract published at `docs/modules/policy/contracts/29-002-streaming-simulation.md`; unblocks 29-003..40-002 chain. |
| `PREP-EXPORT-CONSOLE-23-001` | DOING (2025-11-20) | Drafted export bundle + scheduler job contract (see `docs/modules/policy/design/export-console-bundle-contract.md`); waiting on DSSE/storage decisions from Console/Scheduler/Authority. |
| `PREP-POLICY-AIRGAP-56-001` | DOING (2025-11-20) | Drafted mirror bundle schema for air-gap/ sealed mode (see `docs/modules/policy/design/policy-mirror-bundle-schema.md`); waiting on trust-root and retention policy decisions. |
| `PREP-POLICY-ENGINE-30-001` | DOING (2025-11-20) | Drafted overlay projection contract (see `docs/modules/policy/design/policy-overlay-projection.md`); waiting on 29-004 metrics/log schema from Platform/Observability. |

View File

@@ -0,0 +1,114 @@
# POLICY-ENGINE-29-002 — Streaming Simulation (Path/Scope) Contract
Status: Final · 2025-11-23
Owners: Policy Guild · SBOM Service Guild · Findings Ledger Guild
Working directory: `src/Policy/StellaOps.Policy.Engine`
## Purpose
Define the canonical request/response contract for streaming comparisons between two policy versions with path/scope awareness. Output is explainable per-finding deltas without writes, suitable for Console simulation, Ledger projections, and Advisory AI consumers.
## Request Schema (JSON)
```json
{
"schemaVersion": "1.0.0",
"tenant": "acme", // required
"basePolicyRef": "policy://acme/main@sha256:abcd", // required
"candidatePolicyRef": "policy://acme/feature@sha256:ef01", // required
"subject": {
"purl": "pkg:npm/lodash@4.17.21", // or cpe; at least one
"packagePath": "lib/isEqual.js", // optional, POSIX
"osImage": "ghcr.io/acme/app@sha256:..." // optional
},
"targets": [
{
"filePath": "src/lib/isEqual.js", // required, POSIX
"digest": "c1ab...", // optional hex sha256
"treeDigest": "7ff0...", // optional Merkle root
"pathMatch": "prefix", // exact|prefix|glob
"pattern": "src/lib/", // required with pathMatch
"confidence": 0.92, // 0..1
"depthLimit": 3, // optional int
"evidenceHash": "4f9b...", // optional hex (stable over sorted JSON)
"ingestedAt": "2025-11-20T00:00:00Z", // optional ISO-8601 UTC
"connectorId": "excititor-ghsa" // optional string
}
],
"options": {
"sort": "path,finding,verdict", // required enum; default shown
"maxFindings": 1000, // optional int
"includeTrace": true, // include rule trace fragments
"deterministic": true // must be true; rejects false
}
}
```
- `schemaVersion` pinned to semantic version; breaking changes require bump.
- All string comparisons are case-sensitive except `policy://` refs, which normalise to lowercase host/path.
- At least one `target` required. Each target binds evidence to scope (derived from PREP docs `2025-11-20-policy-engine-29-002-prep.md`).
### Derived Scope Normalisation
For each target:
- `pathMatch` resolution order: `exact` > `prefix` > `glob`; tie-breaker by higher `confidence`, then lexical `filePath`.
- `evidenceHash` = SHA-256 of canonical JSON object with properties sorted ASCII and nulls removed.
## Response Schema (streamed NDJSON)
Each line is a JSON object with stable ordering:
```json
{
"tenant": "acme",
"subject": { "purl": "pkg:npm/lodash@4.17.21", "packagePath": "lib/isEqual.js" },
"target": {
"filePath": "src/lib/isEqual.js",
"pattern": "src/lib/",
"pathMatch": "prefix",
"confidence": 0.92,
"evidenceHash": "4f9b..."
},
"finding": {
"id": "GHSA-35jh-r3h4-6jhm",
"ruleId": "policy.rules.javascript.unsafe-merge",
"severity": "high",
"verdict": {
"base": "deny", // allow|deny|warn|info|not-applicable
"candidate": "warn",
"delta": "softened" // added|removed|hardened|softened|unchanged
},
"evidence": {
"locator": { "filePath": "src/lib/isEqual.js", "digest": "c1ab..." },
"provenance": { "ingestedAt": "2025-11-20T00:00:00Z", "connectorId": "excititor-ghsa" }
}
},
"trace": [
{ "step": "match", "rule": "unsafe-merge", "path": "src/lib/isEqual.js" },
{ "step": "decision", "effect": "warn" }
],
"metrics": {
"evalTicks": 128, // deterministic instruction counter
"rulesEvaluated": 12,
"bindings": 3
}
}
```
- Lines are sorted by `target.filePath`, then `finding.id`, then `finding.ruleId`. No other ordering allowed.
- Absent optional fields must be omitted (not null).
## Error Envelope
Errors surface as NDJSON line with `type: "error"`:
```json
{ "type": "error", "code": "POLICY_29_002_SCHEMA", "message": "target[0].pattern required" }
```
Codes: `POLICY_29_002_SCHEMA`, `POLICY_29_002_UNSUPPORTED_VERSION`, `POLICY_29_002_SCOPE_MISMATCH`, `POLICY_29_002_TOO_MANY_FINDINGS`.
## Determinism Rules
- No wall-clock, RNG, or network access during evaluation.
- All hashes use SHA-256 over canonical JSON (sorted properties, UTF-8, no insignificant whitespace).
- Stable ordering: request-specified sort or default `path,finding,verdict` defined above.
- Trace fragments trimmed to deterministic steps only (no timestamps or hostnames).
## Validation & Conformance
- JSON Schema published at `schemas/policy/29-002-streaming-simulation.schema.json` (to be mirrored in Offline Kit); engines must validate requests before execution.
- CI fixtures: `tests/policy/fixtures/29-002-sample-request.json` and `...-response.ndjson` (to be added when engine implements).
## Compatibility Notes
- Extends PREP docs `2025-11-20-policy-engine-29-002-prep.md` and `2025-11-21-policy-path-scope-29-002-prep.md`; supersedes their draft status.
- Downstream tasks 29-003..40-002 must consume `target` shape above; any change requires bumping `schemaVersion` and updating sprint risks.

View File

@@ -0,0 +1,38 @@
# Surface.Env Package Release Note
Status: Published · 2025-11-23
Owners: Scanner Guild · BuildX Plugin Guild · Ops Guild
Scope: Unblock SURFACE-ENV-03 and BuildX adoption by pinning package version + offline bundle path for `StellaOps.Scanner.Surface.Env`.
## Version & Build Inputs
- **Package ID:** `StellaOps.Scanner.Surface.Env`
- **Version:** `0.1.0-alpha.20251123` (semantic, date-stamped for sprint 136)
- **Source:** `src/Scanner/__Libraries/StellaOps.Scanner.Surface.Env/StellaOps.Scanner.Surface.Env.csproj`
- **Pack command:**
- `dotnet pack src/Scanner/__Libraries/StellaOps.Scanner.Surface.Env/StellaOps.Scanner.Surface.Env.csproj -c Release -o local-nugets /p:Version=0.1.0-alpha.20251123`
- **Restore sources:** `local-nugets/; dotnet-public; nuget.org` (per `Directory.Build.props`).
## Offline / Air-Gap Artefacts
- Copy the produced `.nupkg` to `offline/packages/nugets/StellaOps.Scanner.Surface.Env.0.1.0-alpha.20251123.nupkg`.
- Manifest entry:
- `packageId`: `StellaOps.Scanner.Surface.Env`
- `version`: `0.1.0-alpha.20251123`
- `sha256`: `7f79ec14cc52f0880904eccb6fbc8120bd7d316ab8d6390fef054ed11ee4716e`
- `size`: `14080`
- `createdAt`: `2025-11-23T00:00:00Z`
- No external network calls are required after packing; the offline kit consumes the local file.
## Consumer Guidance
- BuildX plugin (`src/Scanner/StellaOps.Scanner.Sbomer.BuildXPlugin`) should reference `0.1.0-alpha.20251123` via the curated feed `local-nugets/`.
- Scanner WebService/Worker should use the same version once Surface.Env integration tests pass (SCANNER-ENV-02).
- Surface.Validation and Surface.Secrets depend on the env settings; keep prefix defaults and determinism rules from `design/surface-env.md`.
## Verification
- Run `dotnet test` for env library once restore is stable; until then, manual pack is acceptable for BuildX smoke tests.
- Validate package contents:
- Contains `StellaOps.Scanner.Surface.Env.dll` and `StellaOps.Scanner.Surface.Env.xml` docs.
- `lib/net10.0/` target only; no native assets.
- Ensure `local-nugets/` feed lists the package with `nuget list -Source local-nugets` before wiring CI.
## Change Log
- 2025-11-23: Initial release note created to unblock SCANNER-ENV-03 and offline kit wiring; version pinned to `0.1.0-alpha.20251123`.

View File

@@ -0,0 +1,119 @@
# Surface.Secrets Schema (Security-approved)
Status: Final · 2025-11-23
Owners: Security Guild · Scanner Guild · Zastava Guild
Related tasks: SURFACE-SECRETS-01/02/03/04/05/06, SCANNER-SECRETS-03, ZASTAVA-SECRETS-01/02
## Purpose
Canonical schema for declaring and validating secret providers used by Surface consumers (Scanner, Zastava, Scheduler). Supersedes the draft notes in `surface-secrets.md` §45 and unblocks validation + integration tasks.
## Configuration Document
Location: `Surface:Secrets` (appsettings, env, or offline kit manifest). JSON Schema (v2020-12):
```json
{
"$id": "https://stellaops/policy/surface-secrets.schema.json",
"$schema": "https://json-schema.org/draft/2020-12/schema",
"type": "object",
"required": ["provider", "providers"],
"properties": {
"provider": {"type": "string", "enum": ["kubernetes", "file", "inline"], "description": "active provider id"},
"fallbackProvider": {"type": "string", "enum": ["kubernetes", "file", "inline"], "nullable": true},
"providers": {
"type": "object",
"properties": {
"kubernetes": {
"type": "object",
"required": ["namespace", "prefix"],
"properties": {
"namespace": {"type": "string", "pattern": "^[a-z0-9]([-a-z0-9]*[a-z0-9])?$"},
"prefix": {"type": "string", "default": "surface-"},
"ttlSeconds": {"type": "integer", "minimum": 30, "default": 600},
"allowServiceAccountFallback": {"type": "boolean", "default": false}
}
},
"file": {
"type": "object",
"required": ["root"],
"properties": {
"root": {"type": "string", "pattern": "^/.+"},
"format": {"type": "string", "enum": ["json", "yaml"], "default": "json"},
"permissions": {"type": "string", "enum": ["0600"], "default": "0600"}
}
},
"inline": {
"type": "object",
"properties": {
"payloadBase64": {"type": "string", "contentEncoding": "base64"},
"enabled": {"type": "boolean", "default": false}
}
}
},
"additionalProperties": false
},
"requiredSecrets": {
"type": "array",
"items": {
"type": "object",
"required": ["tenant", "component", "secretType"],
"properties": {
"tenant": {"type": "string"},
"component": {"type": "string"},
"secretType": {"type": "string", "enum": ["cas-access", "registry", "attestation", "tls"]},
"name": {"type": "string"}
}
},
"uniqueItems": true
}
},
"additionalProperties": false
}
```
### Secret Object Shape
Secrets resolved by providers must conform to one of:
- `cas-access`: `{ "accessKey": "...", "secretKey": "...", "sessionToken": "..." }`
- `registry`: `{ "username": "...", "password": "..." }` or `{ "token": "..." }`
- `attestation`: `{ "privateKeyPem": "...", "keyId": "...", "rekorApiKey": "..." }`
- `tls`: `{ "certPem": "...", "keyPem": "..." }`
Binary values may be base64 but must decode to UTF-8 cleanly; otherwise return base64 string and mark `isBinary: true` in handle metadata.
## Determinism & Validation
- Deterministic ordering: providers resolved in request order; when multiple secrets share tenant/component/type, choose lexicographically smallest `name` then earliest `ingestedAt` metadata.
- `Surface.Validation` codes: `SURFACE_SECRET_PROVIDER_UNKNOWN`, `SURFACE_SECRET_MISSING`, `SURFACE_SECRET_STALE`, `SURFACE_SECRET_FORMAT_INVALID` (new).
- Validators must reject world-readable files (`permissions != 0600`) and inline payloads when `enabled=false` (air-gapped safety).
## Offline / Air-Gap Profile
- Offline kit must ship `offline/secrets/manifest.json` matching this schema with hashes for each secret blob (SHA-256 hex) and `createdAt` in UTC.
- Importer scripts map manifest entries to file provider layout: `<root>/<tenant>/<component>/<secretType>/<name>.json`.
- No external network calls during validation or resolution.
## Examples
Minimal Kubernetes config:
```json
{
"provider": "kubernetes",
"providers": {"kubernetes": {"namespace": "stellaops-runtime", "prefix": "surface-"}},
"requiredSecrets": [
{"tenant": "acme", "component": "scanner-worker", "secretType": "cas-access"},
{"tenant": "acme", "component": "scanner-worker", "secretType": "registry", "name": "primary"}
]
}
```
File provider (offline):
```json
{
"provider": "file",
"providers": {"file": {"root": "/etc/stellaops/secrets", "format": "json"}},
"requiredSecrets": []
}
```
## Implementation Notes
- Aligns with design doc `surface-secrets.md`; this file is the authoritative schema for SURFACE-SECRETS-01/02/03.
- `ISurfaceSecretProvider` must emit metadata `{ "provider": "kubernetes", "ageDays": <int>, "isBinary": <bool> }` for observability.
- Add JSON Schema to `schemas/surface/surface-secrets.schema.json` (mirrored in Offline Kit) when code lands.
## Acceptance
- SECURITY sign-off requires adherence to this schema and validation rules.
- SURFACE-SECRETS-01 moves to DONE; SURFACE-SECRETS-02 to TODO (implementation); SURFACE-VAL-01 unblocks because schema is defined.

View File

@@ -1,6 +1,6 @@
# Surface.Validation Design (Epic: SURFACE-SHARING)
> **Status:** Draft v1.0 — aligns with tasks `SURFACE-VAL-01..05`, `LANG-SURFACE-01..03`, `ENTRYTRACE-SURFACE-01..02`, `ZASTAVA-SURFACE-02`, `SCANNER-SECRETS-01..03`.
> **Status:** v1.1 (2025-11-23) — aligned to Surface.Secrets schema (`surface-secrets-schema.md`) and Surface.Env release 0.1.0-alpha.20251123; covers tasks `SURFACE-VAL-01..05`, `LANG-SURFACE-01..03`, `ENTRYTRACE-SURFACE-01..02`, `ZASTAVA-SURFACE-02`, `SCANNER-SECRETS-01..03`.
>
> **Audience:** Engineers integrating Surface Env/FS/Secrets, QA guild, Security guild.
@@ -52,6 +52,7 @@ public sealed record SurfaceValidationIssue(
| `SURFACE_ENV_CACHE_DIR_UNWRITABLE` | Error | Cache root not writable or disk full. |
| `SURFACE_SECRET_MISSING` | Error | Secret provider cannot locate required secret type. |
| `SURFACE_SECRET_STALE` | Warning | Secret older than rotation window. |
| `SURFACE_SECRET_FORMAT_INVALID` | Error | Secret payload fails schema validation per `surface-secrets-schema.md`. |
| `SURFACE_FS_ENDPOINT_REACHABILITY` | Error | HEAD request to Surface.FS endpoint failed. |
| `SURFACE_FS_BUCKET_MISMATCH` | Error | Provided bucket does not exist / lacks permissions. |
| `SURFACE_FEATURE_UNKNOWN` | Warning | Feature flag not recognised. |
@@ -71,6 +72,11 @@ services.AddSurfaceValidation(builder =>
Validators can access DI services (e.g., HttpClient, Authority token provider) through the context. To avoid long-running checks, recommended max validation time is 500ms per validator.
### Secrets schema binding
- Validators must load the secrets configuration into the JSON Schema defined in `surface-secrets-schema.md`; reject when provider/fallback ids are outside the allowed set or when file permissions differ from `0600`.
- When `Surface:Secrets:Provider = file`, ensure each required secret exists at `<root>/<tenant>/<component>/<secretType>/<name>.json` with base64 payload matching the secret type contract (see §2 in `surface-secrets-schema.md`).
- Inline provider must be disabled in production (`AllowInline=false`); validation emits `SURFACE_SECRET_FORMAT_INVALID` if enabled without an explicit dev/test flag.
## 5. Reporting & Observability
- Results exposed via `ISurfaceValidationReporter` (default logs structured JSON to `Validation` category).