This commit is contained in:
139
docs/export-center/profiles.md
Normal file
139
docs/export-center/profiles.md
Normal file
@@ -0,0 +1,139 @@
|
||||
# Export Center Profiles
|
||||
|
||||
Export Center profiles define what data is collected, how it is encoded, and which distribution paths are enabled for a run. Profiles are tenant-scoped and deterministic: identical selectors and source data produce identical bundles. This guide summarises built-in profiles, configuration fields, schema conventions, and compatibility notes.
|
||||
|
||||
## Profile catalogue
|
||||
|
||||
| Profile | Kind / Variant | Output artefacts | Primary use cases |
|
||||
|---------|----------------|------------------|-------------------|
|
||||
| `json:raw` | `json` / `raw` | Canonical JSONL archives of advisories, VEX, SBOMs | Evidence escrow, analytics pipelines |
|
||||
| `json:policy` | `json` / `policy` | `json:raw` artefacts plus policy snapshot + evaluated findings | Audit, compliance attestations |
|
||||
| `trivy:db` | `trivy` / `db` | Trivy-compatible vulnerability database | Feeding external scanners / CI |
|
||||
| `trivy:java-db` | `trivy` / `java-db` | Java ecosystem supplement for Trivy | Supply Java CVE data to Trivy |
|
||||
| `mirror:full` | `mirror` / `full` | Complete mirror bundle (raw, policy, indexes, provenance) | Air-gap deployments, disaster recovery |
|
||||
| `mirror:delta` | `mirror` / `delta` | Incremental changes relative to a prior manifest | Efficient mirror updates |
|
||||
|
||||
Profiles can be cloned and customised; configuration is immutable per revision to keep runs reproducible.
|
||||
|
||||
## Common configuration fields
|
||||
|
||||
| Field | Description | Applies to | Notes |
|
||||
|-------|-------------|------------|-------|
|
||||
| `name` | Human-readable identifier displayed in Console/CLI | All | Unique per tenant. |
|
||||
| `kind` | Logical family (`json`, `trivy`, `mirror`) | All | Determines eligible adapters. |
|
||||
| `variant` | Specific export flavour (see table above) | All | Controls adapter behaviour. |
|
||||
| `include` | Record types to include (`advisories`, `vex`, `sboms`, `findings`) | JSON, mirror | Defaults depend on variant. |
|
||||
| `policySnapshotMode` | `required`, `optional`, or `none` | JSON policy, mirror | `required` forces a policy snapshot id when creating runs. |
|
||||
| `distribution` | Enabled distribution drivers (`http`, `oci`, `object`) | All | Offline installs typically disable `oci`. |
|
||||
| `compression` | Compression settings (`zstd`, level) | JSON, mirror | Trivy adapters manage compression internally. |
|
||||
| `encryption` | Mirror encryption options (`enabled`, `recipientKeys`, `strict`) | Mirror | When enabled, only `/data` subtree is encrypted; manifests remain plaintext. |
|
||||
| `retention` | Retention policy (days or `never`) | All | Drives pruning jobs for staged bundles. |
|
||||
|
||||
Selectors (time windows, tenants, products, SBOM subjects, ecosystems) are supplied per run, not stored in the profile.
|
||||
|
||||
## JSON profiles
|
||||
|
||||
### `json:raw`
|
||||
- **Content:** Exports raw advisories, VEX statements, and SBOMs as newline-delimited JSON (`.jsonl.zst`).
|
||||
- **Schema:** Follows canonical StellaOps schema with casing and timestamps normalised. Each record includes `tenant`, `source`, `linkset`, and `content` fields.
|
||||
- **Options:**
|
||||
- `include` defaults to `["advisories", "vex", "sboms"]`.
|
||||
- `compression` defaults to `zstd` level 9.
|
||||
- `maxRecordsPerFile` (optional) splits outputs for large datasets.
|
||||
- **Compatibility:** Intended for analytics platforms, data escrow, or feeding downstream normalisation pipelines.
|
||||
- **Sample manifest excerpt:**
|
||||
|
||||
```json
|
||||
{
|
||||
"profile": { "kind": "json", "variant": "raw" },
|
||||
"outputs": [
|
||||
{ "type": "advisories.jsonl.zst", "sha256": "...", "count": 15234 },
|
||||
{ "type": "vex.jsonl.zst", "sha256": "...", "count": 3045 },
|
||||
{ "type": "sboms.jsonl.zst", "sha256": "...", "count": 872 }
|
||||
],
|
||||
"selectors": { "tenant": "acme", "products": ["registry.example/app"] }
|
||||
}
|
||||
```
|
||||
|
||||
### `json:policy`
|
||||
- **Content:** Everything from `json:raw` plus:
|
||||
- `policy_snapshot.json` (policy metadata, version, hash).
|
||||
- `findings.policy.jsonl.zst` (evaluated findings with decision, rationale, rule id, inputs hash).
|
||||
- **Determinism:** Requires a policy snapshot id; runs fail if snapshot is missing or non-deterministic mode is active.
|
||||
- **Use cases:** Compliance exports, auditor packages, policy attestation archives.
|
||||
- **Guardrails:** AOC boundaries preserved: policy outputs are clearly partitioned from raw evidence.
|
||||
|
||||
## Trivy profiles
|
||||
|
||||
### `trivy:db`
|
||||
- Detailed adapter behaviour is documented in `docs/export-center/trivy-adapter.md`.
|
||||
- **Content:** Produces a Trivy DB-compatible bundle (SQLite database or tarball as required by Trivy version).
|
||||
- **Mapping rules:**
|
||||
- Advisory namespaces mapped to Trivy vendor IDs (e.g., `ubuntu`, `debian`, `npm`).
|
||||
- Version ranges translated into Trivy's semantic version syntax.
|
||||
- Severity mapped to Trivy standard (`UNKNOWN`, `LOW`, `MEDIUM`, `HIGH`, `CRITICAL`).
|
||||
- **Validation:** Adapter enforces supported Trivy schema versions; configuration includes `targetSchemaVersion`.
|
||||
- **Distribution:** Typically pushed to OCI or object storage for downstream scanners; Console download remains available.
|
||||
|
||||
### `trivy:java-db`
|
||||
- Refer to `docs/export-center/trivy-adapter.md` for ecosystem-specific notes.
|
||||
- **Content:** Optional Java ecosystem supplement for Trivy (matching Trivy's separate Java DB).
|
||||
- **Dependencies:** Requires Java advisories in Findings Ledger; run fails with `ERR_EXPORT_EMPTY` if no Java data present and `allowEmpty=false`.
|
||||
- **Compatibility:** Intended for organisations using Trivy's Java plugin or hardened pipelines that split general and Java feeds.
|
||||
|
||||
## Mirror profiles
|
||||
|
||||
### `mirror:full`
|
||||
- Bundle structure and delta strategy are covered in `docs/export-center/mirror-bundles.md`.
|
||||
- **Content:** Complete export with:
|
||||
- Raw advisories, VEX, SBOMs (`/data/raw`).
|
||||
- Policy overlays (`/data/policy`), including evaluated findings and policy snapshots.
|
||||
- Indexes for fast lookup (`/indexes/advisories.pb`, `/indexes/sboms.pb`).
|
||||
- Manifests and provenance (`/manifests/export.json`, `/manifests/provenance.json`).
|
||||
- **Layout:** Deterministic directory structure with hashed filenames to reduce duplication.
|
||||
- **Encryption:** Optional `encryption` block enables age/AES-GCM encryption of `/data`. `strict=true` encrypts everything except `export.json`.
|
||||
- **Use cases:** Air-gap replication, disaster recovery drills, Offline Kit seeding.
|
||||
|
||||
### `mirror:delta`
|
||||
- See `docs/export-center/mirror-bundles.md` for delta mechanics and application order.
|
||||
- **Content:** Includes only changes relative to a base manifest (specified by `baseExportId` when running).
|
||||
- `changed`, `added`, `removed` lists in `manifests/delta.json`.
|
||||
- Incremental indexes capturing only updated subjects.
|
||||
- **Constraints:** Requires the base manifest to exist in object storage or artifact registry accessible to the worker. Fails with `ERR_EXPORT_BASE_MISSING` otherwise.
|
||||
- **Workflow:** Ideal for frequent updates to mirrored environments with limited bandwidth.
|
||||
|
||||
## Compatibility and guardrails
|
||||
- **Aggregation-Only Contract:** All profiles respect AOC boundaries: raw evidence is never mutated. Policy outputs are appended separately with clear provenance.
|
||||
- **Tenant scoping:** Profiles are tenant-specific. Cross-tenant exports require explicit administrative approval and signed justification.
|
||||
- **Retriable runs:** Re-running a profile with identical selectors yields matching manifests and hashes, facilitating verify-on-download workflows.
|
||||
- **Offline operation:** JSON and mirror profiles function in offline mode without additional configuration. Trivy profiles require pre-seeded schema metadata shipped via Offline Kit.
|
||||
- **Quota integration:** Profiles can define run quotas (per tenant per day). Quota exhaustion surfaces as `429 Too Many Requests` with `X-Stella-Quota-*` hints.
|
||||
|
||||
## Example profile definition (CLI)
|
||||
|
||||
```jsonc
|
||||
{
|
||||
"name": "daily-json-raw",
|
||||
"kind": "json",
|
||||
"variant": "raw",
|
||||
"include": ["advisories", "vex", "sboms"],
|
||||
"distribution": ["http", "object"],
|
||||
"compression": { "codec": "zstd", "level": 9 },
|
||||
"retention": { "mode": "days", "value": 14 }
|
||||
}
|
||||
```
|
||||
|
||||
Create via `stella export profile create --file profile.json` (CLI command documented separately).
|
||||
|
||||
## Verification workflow
|
||||
- Download bundle via Console/CLI and extract `export.json` and `provenance.json`.
|
||||
- Run `cosign verify --key <tenant-key> export.json` (for detatched signatures use `--signature export.json.sig`).
|
||||
- Validate Trivy bundles with `trivy --cache-dir <temp> --debug --db-repository <oci-ref>` or local `trivy module db import`.
|
||||
- For mirror bundles, run internal `mirror verify` script (bundled in Offline Kit) to ensure directory layout and digests match manifest.
|
||||
|
||||
## Extending profiles
|
||||
- Use API/CLI to clone an existing profile and adjust `include`, `distribution`, or retention policies.
|
||||
- Adapter plug-ins can introduce new variants (e.g., `json:raw-lite`, `mirror:policy-only`). Plug-ins must be registered at service restart and documented under `/docs/export-center/profiles.md`.
|
||||
- Any new profile must append the imposed rule line and follow determinism and guardrail requirements.
|
||||
|
||||
> **Imposed rule:** Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.
|
||||
Reference in New Issue
Block a user