tests fixes and sprints work
This commit is contained in:
master
@@ -23,10 +23,12 @@ The `stella` CLI is the operator-facing Swiss army knife for scans, exports, pol
|
||||
- Versioned command docs in `docs/modules/cli/guides`.
|
||||
- Plugin catalogue in `plugins/cli/**` (restart-only).
|
||||
|
||||
## Related resources
|
||||
- ./guides/20_REFERENCE.md
|
||||
- ./guides/cli-reference.md
|
||||
- ./guides/policy.md
|
||||
## Related resources
|
||||
- ./guides/20_REFERENCE.md
|
||||
- ./guides/cli-reference.md
|
||||
- ./guides/commands/analytics.md
|
||||
- ./guides/policy.md
|
||||
- ./guides/trust-profiles.md
|
||||
|
||||
## Backlog references
|
||||
- DOCS-CLI-OBS-52-001 / DOCS-CLI-FORENSICS-53-001 in ../../TASKS.md.
|
||||
|
||||
@@ -51,10 +51,11 @@ Status key:
|
||||
|
||||
| UI capability | CLI command(s) | Status | Notes / Tasks |
|
||||
|---------------|----------------|--------|---------------|
|
||||
| Advisory observations search | `stella vuln observations` | ✅ Available | Implemented via `BuildVulnCommand`. |
|
||||
| Advisory linkset export | `stella advisory linkset show/export` | 🟩 Planned | `CLI-LNM-22-001`. |
|
||||
| VEX observations / linksets | `stella vex obs get/linkset show` | 🟩 Planned | `CLI-LNM-22-002`. |
|
||||
| SBOM overlay export | `stella sbom overlay apply/export` | 🟩 Planned | Scoped to upcoming SBOM CLI sprint (`SBOM-CONSOLE-23-001/002` + CLI backlog). |
|
||||
| Advisory observations search | `stella vuln observations` | ✅ Available | Implemented via `BuildVulnCommand`. |
|
||||
| Advisory linkset export | `stella advisory linkset show/export` | 🟩 Planned | `CLI-LNM-22-001`. |
|
||||
| VEX observations / linksets | `stella vex obs get/linkset show` | 🟩 Planned | `CLI-LNM-22-002`. |
|
||||
| SBOM overlay export | `stella sbom overlay apply/export` | 🟩 Planned | Scoped to upcoming SBOM CLI sprint (`SBOM-CONSOLE-23-001/002` + CLI backlog). |
|
||||
| SBOM Lake analytics (`/analytics/sbom-lake`) | `stella analytics sbom-lake <subcommand>` | ✅ Available | CLI guide at `docs/modules/cli/guides/commands/analytics.md` (SPRINT_20260120_032). |
|
||||
|
||||
---
|
||||
|
||||
@@ -151,5 +152,5 @@ The script should emit a parity report that feeds into the Downloads workspace (
|
||||
|
||||
---
|
||||
|
||||
*Last updated: 2025-10-28 (Sprint 23).*
|
||||
*Last updated: 2026-01-20 (Sprint 20260120).*
|
||||
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
version: 1
|
||||
generated: 2025-12-01T00:00:00Z
|
||||
generated: 2026-01-20T00:00:00Z
|
||||
compatibility:
|
||||
policy: "SemVer-like: commands/flags/exitCodes are backwards compatible within major version."
|
||||
deprecation:
|
||||
@@ -38,6 +38,108 @@ commands:
|
||||
0: success
|
||||
4: auth-misconfigured
|
||||
5: token-invalid
|
||||
- name: analytics
|
||||
subcommands:
|
||||
- name: sbom-lake
|
||||
subcommands:
|
||||
- name: suppliers
|
||||
formats: [table, json, csv]
|
||||
flags:
|
||||
- name: environment
|
||||
required: false
|
||||
- name: limit
|
||||
required: false
|
||||
- name: format
|
||||
required: false
|
||||
values: [table, json, csv]
|
||||
- name: output
|
||||
required: false
|
||||
exitCodes:
|
||||
0: success
|
||||
1: error
|
||||
- name: licenses
|
||||
formats: [table, json, csv]
|
||||
flags:
|
||||
- name: environment
|
||||
required: false
|
||||
- name: limit
|
||||
required: false
|
||||
- name: format
|
||||
required: false
|
||||
values: [table, json, csv]
|
||||
- name: output
|
||||
required: false
|
||||
exitCodes:
|
||||
0: success
|
||||
1: error
|
||||
- name: vulnerabilities
|
||||
formats: [table, json, csv]
|
||||
flags:
|
||||
- name: environment
|
||||
required: false
|
||||
- name: min-severity
|
||||
required: false
|
||||
values: [critical, high, medium, low]
|
||||
- name: limit
|
||||
required: false
|
||||
- name: format
|
||||
required: false
|
||||
values: [table, json, csv]
|
||||
- name: output
|
||||
required: false
|
||||
exitCodes:
|
||||
0: success
|
||||
1: error
|
||||
- name: backlog
|
||||
formats: [table, json, csv]
|
||||
flags:
|
||||
- name: environment
|
||||
required: false
|
||||
- name: limit
|
||||
required: false
|
||||
- name: format
|
||||
required: false
|
||||
values: [table, json, csv]
|
||||
- name: output
|
||||
required: false
|
||||
exitCodes:
|
||||
0: success
|
||||
1: error
|
||||
- name: attestation-coverage
|
||||
formats: [table, json, csv]
|
||||
flags:
|
||||
- name: environment
|
||||
required: false
|
||||
- name: limit
|
||||
required: false
|
||||
- name: format
|
||||
required: false
|
||||
values: [table, json, csv]
|
||||
- name: output
|
||||
required: false
|
||||
exitCodes:
|
||||
0: success
|
||||
1: error
|
||||
- name: trends
|
||||
formats: [table, json, csv]
|
||||
flags:
|
||||
- name: environment
|
||||
required: false
|
||||
- name: days
|
||||
required: false
|
||||
- name: series
|
||||
required: false
|
||||
values: [vulnerabilities, components, all]
|
||||
- name: limit
|
||||
required: false
|
||||
- name: format
|
||||
required: false
|
||||
values: [table, json, csv]
|
||||
- name: output
|
||||
required: false
|
||||
exitCodes:
|
||||
0: success
|
||||
1: error
|
||||
telemetry:
|
||||
defaultEnabled: false
|
||||
envVars:
|
||||
|
||||
47
docs/modules/cli/guides/commands/analytics.md
Normal file
47
docs/modules/cli/guides/commands/analytics.md
Normal file
@@ -0,0 +1,47 @@
|
||||
# stella analytics - Command Guide
|
||||
|
||||
## Commands
|
||||
- `stella analytics sbom-lake suppliers [--environment <env>] [--limit <n>] [--format table|json|csv] [--output <path>]`
|
||||
- `stella analytics sbom-lake licenses [--environment <env>] [--limit <n>] [--format table|json|csv] [--output <path>]`
|
||||
- `stella analytics sbom-lake vulnerabilities [--environment <env>] [--min-severity <level>] [--limit <n>] [--format table|json|csv] [--output <path>]`
|
||||
- `stella analytics sbom-lake backlog [--environment <env>] [--limit <n>] [--format table|json|csv] [--output <path>]`
|
||||
- `stella analytics sbom-lake attestation-coverage [--environment <env>] [--limit <n>] [--format table|json|csv] [--output <path>]`
|
||||
- `stella analytics sbom-lake trends [--environment <env>] [--days <n>] [--series vulnerabilities|components|all] [--limit <n>] [--format table|json|csv] [--output <path>]`
|
||||
|
||||
## Flags (common)
|
||||
- `--format`: Output format for rendering (`table`, `json`, `csv`).
|
||||
- `--output`: Write output to a file path instead of stdout.
|
||||
- `--limit`: Cap the number of rows returned.
|
||||
- `--environment`: Filter by environment name.
|
||||
|
||||
## SBOM lake notes
|
||||
- Endpoints require the `analytics.read` scope.
|
||||
- `--min-severity` accepts `critical`, `high`, `medium`, `low`.
|
||||
- `--series` controls trend output (`vulnerabilities`, `components`, `all`).
|
||||
- Tables use deterministic ordering (severity and counts first, then names).
|
||||
|
||||
## Examples
|
||||
|
||||
```bash
|
||||
# Top suppliers
|
||||
stella analytics sbom-lake suppliers --limit 20
|
||||
|
||||
# License distribution as CSV (prod)
|
||||
stella analytics sbom-lake licenses --environment prod --format csv --output licenses.csv
|
||||
|
||||
# Vulnerability exposure in prod (high+)
|
||||
stella analytics sbom-lake vulnerabilities --environment prod --min-severity high
|
||||
|
||||
# Fixable backlog with table output
|
||||
stella analytics sbom-lake backlog --environment prod --limit 50
|
||||
|
||||
# Attestation coverage in staging, JSON output
|
||||
stella analytics sbom-lake attestation-coverage --environment stage --format json
|
||||
|
||||
# 30-day trend snapshot (both series)
|
||||
stella analytics sbom-lake trends --days 30 --series all --format csv --output trends.csv
|
||||
```
|
||||
|
||||
## Offline/verification note
|
||||
- If analytics exports arrive via offline bundles, verify the bundle first with
|
||||
`stella bundle verify` before importing data into downstream reports.
|
||||
@@ -16,6 +16,7 @@ graph TD
|
||||
CLI --> EXPLAIN[Explainability]
|
||||
CLI --> VEX[VEX & Decisioning]
|
||||
CLI --> SBOM[SBOM Operations]
|
||||
CLI --> ANALYTICS[Analytics & Insights]
|
||||
CLI --> REPORT[Reporting & Export]
|
||||
CLI --> OFFLINE[Offline Operations]
|
||||
CLI --> SYSTEM[System & Config]
|
||||
@@ -742,6 +743,601 @@ stella sbom merge --sbom <path1> --sbom <path2> [--output <path>] [--verbose]
|
||||
|
||||
---
|
||||
|
||||
## Analytics Commands
|
||||
|
||||
### stella analytics sbom-lake
|
||||
|
||||
Query SBOM lake analytics views (suppliers, licenses, vulnerabilities, backlog,
|
||||
attestation coverage, trends).
|
||||
|
||||
**Usage:**
|
||||
```bash
|
||||
stella analytics sbom-lake <subcommand> [options]
|
||||
```
|
||||
|
||||
**Subcommands:**
|
||||
- `suppliers` - Supplier concentration
|
||||
- `licenses` - License distribution
|
||||
- `vulnerabilities` - CVE exposure (VEX-adjusted)
|
||||
- `backlog` - Fixable vulnerability backlog
|
||||
- `attestation-coverage` - Provenance/SLSA coverage
|
||||
- `trends` - Time-series trends (vulnerabilities/components)
|
||||
|
||||
**Common options:**
|
||||
| Option | Description |
|
||||
|--------|-------------|
|
||||
| `--environment <env>` | Filter to a specific environment |
|
||||
| `--min-severity <level>` | Minimum severity (`critical`, `high`, `medium`, `low`) |
|
||||
| `--days <n>` | Lookback window in days (trends only) |
|
||||
| `--series <name>` | Trend series (`vulnerabilities`, `components`, `all`) |
|
||||
| `--limit <n>` | Maximum number of rows |
|
||||
| `--format <fmt>` | Output format: `table`, `json`, `csv` |
|
||||
| `--output <path>` | Output file path |
|
||||
|
||||
**Example:**
|
||||
```bash
|
||||
stella analytics sbom-lake vulnerabilities --environment prod --min-severity high --format csv --output vuln.csv
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Ground-Truth Corpus Commands
|
||||
|
||||
### stella groundtruth
|
||||
|
||||
Manage ground-truth corpus for patch-paired binary verification. The corpus supports
|
||||
precision validation of security advisories by maintaining symbol and binary pairs
|
||||
from upstream sources.
|
||||
|
||||
**Sprint:** SPRINT_20260121_035_BinaryIndex_golden_corpus_connectors_cli
|
||||
|
||||
**Usage:**
|
||||
```bash
|
||||
stella groundtruth <subcommand> [options]
|
||||
```
|
||||
|
||||
**Subcommands:**
|
||||
- `sources` - Manage symbol source connectors
|
||||
- `symbols` - Query and search symbols in the corpus
|
||||
- `pairs` - Manage security pairs (vuln/patch binary pairs)
|
||||
- `validate` - Run validation and view metrics
|
||||
|
||||
---
|
||||
|
||||
### stella groundtruth sources
|
||||
|
||||
Manage upstream symbol source connectors.
|
||||
|
||||
**Usage:**
|
||||
```bash
|
||||
stella groundtruth sources <command> [options]
|
||||
```
|
||||
|
||||
**Subcommands:**
|
||||
|
||||
#### stella groundtruth sources list
|
||||
|
||||
List available symbol source connectors.
|
||||
|
||||
```bash
|
||||
stella groundtruth sources list [--output-format table|json] [--verbose]
|
||||
```
|
||||
|
||||
**Output:**
|
||||
```
|
||||
ID Display Name Status Last Sync
|
||||
------------------------------------------------------------------------------------------
|
||||
debuginfod-fedora Fedora Debuginfod Enabled 2026-01-22T10:00:00Z
|
||||
debuginfod-ubuntu Ubuntu Debuginfod Enabled 2026-01-22T10:00:00Z
|
||||
ddeb-ubuntu Ubuntu ddebs Enabled 2026-01-22T09:30:00Z
|
||||
buildinfo-debian Debian Buildinfo Enabled 2026-01-22T08:00:00Z
|
||||
secdb-alpine Alpine SecDB Enabled 2026-01-22T06:00:00Z
|
||||
```
|
||||
|
||||
#### stella groundtruth sources enable
|
||||
|
||||
Enable a symbol source connector.
|
||||
|
||||
```bash
|
||||
stella groundtruth sources enable <source> [--verbose]
|
||||
```
|
||||
|
||||
**Arguments:**
|
||||
- `<source>` - Source connector ID (e.g., `debuginfod-fedora`)
|
||||
|
||||
**Example:**
|
||||
```bash
|
||||
stella groundtruth sources enable debuginfod-fedora
|
||||
```
|
||||
|
||||
#### stella groundtruth sources disable
|
||||
|
||||
Disable a symbol source connector.
|
||||
|
||||
```bash
|
||||
stella groundtruth sources disable <source> [--verbose]
|
||||
```
|
||||
|
||||
#### stella groundtruth sources sync
|
||||
|
||||
Synchronize symbol sources from upstream.
|
||||
|
||||
```bash
|
||||
stella groundtruth sources sync [--source <id>] [--full] [--verbose]
|
||||
```
|
||||
|
||||
**Options:**
|
||||
| Option | Description |
|
||||
|--------|-------------|
|
||||
| `--source <id>` | Source connector ID (all if not specified) |
|
||||
| `--full` | Perform a full sync instead of incremental |
|
||||
|
||||
**Example:**
|
||||
```bash
|
||||
# Incremental sync of all sources
|
||||
stella groundtruth sources sync
|
||||
|
||||
# Full sync of Debian buildinfo
|
||||
stella groundtruth sources sync --source buildinfo-debian --full
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### stella groundtruth symbols
|
||||
|
||||
Query and search symbols in the corpus.
|
||||
|
||||
**Usage:**
|
||||
```bash
|
||||
stella groundtruth symbols <command> [options]
|
||||
```
|
||||
|
||||
#### stella groundtruth symbols lookup
|
||||
|
||||
Lookup symbols by debug ID (build-id).
|
||||
|
||||
```bash
|
||||
stella groundtruth symbols lookup --debug-id <id> [--output-format table|json] [--verbose]
|
||||
```
|
||||
|
||||
**Options:**
|
||||
| Option | Alias | Description | Required |
|
||||
|--------|-------|-------------|----------|
|
||||
| `--debug-id` | `-d` | Debug ID (build-id) to lookup | Yes |
|
||||
| `--output-format` | `-O` | Output format: `table`, `json` | No |
|
||||
|
||||
**Example:**
|
||||
```bash
|
||||
stella groundtruth symbols lookup --debug-id 7f8a9b2c4d5e6f1a --output-format json
|
||||
```
|
||||
|
||||
**Output (table):**
|
||||
```
|
||||
Binary: libcrypto.so.3
|
||||
Architecture: x86_64
|
||||
Distribution: debian-bookworm
|
||||
Package: openssl@3.0.11-1
|
||||
Symbol Count: 4523
|
||||
Sources: debuginfod-fedora, buildinfo-debian
|
||||
```
|
||||
|
||||
#### stella groundtruth symbols search
|
||||
|
||||
Search symbols by package or distribution.
|
||||
|
||||
```bash
|
||||
stella groundtruth symbols search [--package <name>] [--distro <distro>] [--limit <n>] [--output-format table|json] [--verbose]
|
||||
```
|
||||
|
||||
**Options:**
|
||||
| Option | Alias | Description | Default |
|
||||
|--------|-------|-------------|---------|
|
||||
| `--package` | `-p` | Package name to search for | - |
|
||||
| `--distro` | | Distribution filter (debian, ubuntu, alpine) | - |
|
||||
| `--limit` | `-l` | Maximum results | 20 |
|
||||
|
||||
**Example:**
|
||||
```bash
|
||||
stella groundtruth symbols search --package openssl --distro debian --limit 50
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### stella groundtruth pairs
|
||||
|
||||
Manage security pairs (vulnerable/patched binary pairs) in the corpus.
|
||||
|
||||
**Usage:**
|
||||
```bash
|
||||
stella groundtruth pairs <command> [options]
|
||||
```
|
||||
|
||||
#### stella groundtruth pairs create
|
||||
|
||||
Create a new security pair.
|
||||
|
||||
```bash
|
||||
stella groundtruth pairs create --cve <cve-id> --vuln-pkg <pkg=ver> --patch-pkg <pkg=ver> [--distro <distro>] [--verbose]
|
||||
```
|
||||
|
||||
**Options:**
|
||||
| Option | Description | Required |
|
||||
|--------|-------------|----------|
|
||||
| `--cve` | CVE identifier | Yes |
|
||||
| `--vuln-pkg` | Vulnerable package (name=version) | Yes |
|
||||
| `--patch-pkg` | Patched package (name=version) | Yes |
|
||||
| `--distro` | Distribution (e.g., `debian-bookworm`) | No |
|
||||
|
||||
**Example:**
|
||||
```bash
|
||||
stella groundtruth pairs create \
|
||||
--cve CVE-2024-1234 \
|
||||
--vuln-pkg openssl=3.0.10-1 \
|
||||
--patch-pkg openssl=3.0.11-1 \
|
||||
--distro debian-bookworm
|
||||
```
|
||||
|
||||
#### stella groundtruth pairs list
|
||||
|
||||
List security pairs in the corpus.
|
||||
|
||||
```bash
|
||||
stella groundtruth pairs list [--cve <pattern>] [--package <name>] [--limit <n>] [--output-format table|json] [--verbose]
|
||||
```
|
||||
|
||||
**Options:**
|
||||
| Option | Alias | Description | Default |
|
||||
|--------|-------|-------------|---------|
|
||||
| `--cve` | | Filter by CVE (supports wildcards: `CVE-2024-*`) | - |
|
||||
| `--package` | `-p` | Filter by package name | - |
|
||||
| `--limit` | `-l` | Maximum results | 50 |
|
||||
|
||||
**Example:**
|
||||
```bash
|
||||
stella groundtruth pairs list --cve CVE-2024-* --package openssl --limit 100
|
||||
```
|
||||
|
||||
**Output:**
|
||||
```
|
||||
Pair ID CVE Package Vuln Version Patch Version
|
||||
-------------------------------------------------------------------------------
|
||||
pair-001 CVE-2024-1234 openssl 3.0.10-1 3.0.11-1
|
||||
pair-002 CVE-2024-5678 curl 8.4.0-1 8.5.0-1
|
||||
```
|
||||
|
||||
#### stella groundtruth pairs delete
|
||||
|
||||
Delete a security pair from the corpus.
|
||||
|
||||
```bash
|
||||
stella groundtruth pairs delete <pair-id> [--force] [--verbose]
|
||||
```
|
||||
|
||||
**Options:**
|
||||
| Option | Alias | Description |
|
||||
|--------|-------|-------------|
|
||||
| `--force` | `-f` | Skip confirmation prompt |
|
||||
|
||||
---
|
||||
|
||||
### stella groundtruth validate
|
||||
|
||||
Run validation harness against security pairs.
|
||||
|
||||
**Usage:**
|
||||
```bash
|
||||
stella groundtruth validate <command> [options]
|
||||
```
|
||||
|
||||
#### stella groundtruth validate run
|
||||
|
||||
Run validation on security pairs.
|
||||
|
||||
```bash
|
||||
stella groundtruth validate run [--pairs <pattern>] [--matcher <type>] [--output <path>] [--parallel <n>] [--verbose]
|
||||
```
|
||||
|
||||
**Options:**
|
||||
| Option | Alias | Description | Default |
|
||||
|--------|-------|-------------|---------|
|
||||
| `--pairs` | `-p` | Pair filter pattern (e.g., `openssl:CVE-2024-*`) | all |
|
||||
| `--matcher` | `-m` | Matcher type: `semantic-diffing`, `hash-based`, `hybrid` | `semantic-diffing` |
|
||||
| `--output` | `-o` | Output file for validation report | - |
|
||||
| `--parallel` | | Maximum parallel validations | 4 |
|
||||
|
||||
**Example:**
|
||||
```bash
|
||||
stella groundtruth validate run \
|
||||
--pairs "openssl:CVE-2024-*" \
|
||||
--matcher semantic-diffing \
|
||||
--parallel 8 \
|
||||
--output validation-report.md
|
||||
```
|
||||
|
||||
**Output:**
|
||||
```
|
||||
Validating pairs: 10/10
|
||||
Validation complete. Run ID: vr-20260122100532
|
||||
Function Match Rate: 94.2%
|
||||
False-Negative Rate: 2.1%
|
||||
SBOM Hash Stability: 3/3
|
||||
Report written to: validation-report.md
|
||||
```
|
||||
|
||||
#### stella groundtruth validate metrics
|
||||
|
||||
View metrics for a validation run.
|
||||
|
||||
```bash
|
||||
stella groundtruth validate metrics --run-id <id> [--output-format table|json] [--verbose]
|
||||
```
|
||||
|
||||
**Options:**
|
||||
| Option | Alias | Description | Required |
|
||||
|--------|-------|-------------|----------|
|
||||
| `--run-id` | `-r` | Validation run ID | Yes |
|
||||
|
||||
**Example:**
|
||||
```bash
|
||||
stella groundtruth validate metrics --run-id vr-20260122100532 --output-format json
|
||||
```
|
||||
|
||||
**Output (table):**
|
||||
```
|
||||
Run ID: vr-20260122100532
|
||||
Duration: 2026-01-22T10:00:00Z - 2026-01-22T10:15:32Z
|
||||
Pairs: 48/50 successful
|
||||
Function Match Rate: 94.2%
|
||||
False-Negative Rate: 2.1%
|
||||
SBOM Hash Stability: 3/3
|
||||
Verify Time (p50/p95): 423ms / 1.2s
|
||||
```
|
||||
|
||||
#### stella groundtruth validate export
|
||||
|
||||
Export validation report.
|
||||
|
||||
```bash
|
||||
stella groundtruth validate export --run-id <id> --output <path> [--format <fmt>] [--verbose]
|
||||
```
|
||||
|
||||
**Options:**
|
||||
| Option | Alias | Description | Default |
|
||||
|--------|-------|-------------|---------|
|
||||
| `--run-id` | `-r` | Validation run ID | (required) |
|
||||
| `--output` | `-o` | Output file path | (required) |
|
||||
| `--format` | `-f` | Export format: `markdown`, `html`, `json` | `markdown` |
|
||||
|
||||
**Example:**
|
||||
```bash
|
||||
stella groundtruth validate export \
|
||||
--run-id vr-20260122100532 \
|
||||
--format markdown \
|
||||
--output validation-report.md
|
||||
```
|
||||
|
||||
**See Also:** [Ground-Truth CLI Guide](../ground-truth-cli.md)
|
||||
|
||||
---
|
||||
|
||||
### stella groundtruth bundle
|
||||
|
||||
Manage evidence bundles for offline verification of patch provenance.
|
||||
|
||||
**Sprint:** SPRINT_20260121_036_BinaryIndex_golden_corpus_bundle_verification
|
||||
|
||||
**Usage:**
|
||||
```bash
|
||||
stella groundtruth bundle <command> [options]
|
||||
```
|
||||
|
||||
**Subcommands:**
|
||||
- `export` - Create evidence bundles for air-gapped environments
|
||||
- `import` - Import and verify evidence bundles
|
||||
|
||||
#### stella groundtruth bundle export
|
||||
|
||||
Export evidence bundles containing pre/post binaries, SBOMs, delta-sig predicates, and timestamps.
|
||||
|
||||
```bash
|
||||
stella groundtruth bundle export [options]
|
||||
```
|
||||
|
||||
**Options:**
|
||||
| Option | Description | Required |
|
||||
|--------|-------------|----------|
|
||||
| `--packages <list>` | Comma-separated package names (e.g., `openssl,curl`) | Yes |
|
||||
| `--distros <list>` | Comma-separated distributions (e.g., `debian,ubuntu`) | Yes |
|
||||
| `--output <path>` | Output bundle path (.tar.gz or .oci.tar) | Yes |
|
||||
| `--sign-with <signer>` | Signing method: `cosign`, `sigstore`, `none` | No |
|
||||
| `--include-debug` | Include debug symbols | No |
|
||||
| `--include-kpis` | Include KPI validation results | No |
|
||||
| `--include-timestamps` | Include RFC 3161 timestamps | No |
|
||||
|
||||
**Example:**
|
||||
```bash
|
||||
stella groundtruth bundle export \
|
||||
--packages openssl,zlib,glibc \
|
||||
--distros debian,fedora \
|
||||
--output evidence/security-bundle.tar.gz \
|
||||
--sign-with cosign \
|
||||
--include-debug \
|
||||
--include-kpis \
|
||||
--include-timestamps
|
||||
```
|
||||
|
||||
**Exit Codes:**
|
||||
- `0` - Bundle created successfully
|
||||
- `1` - Bundle creation failed
|
||||
- `2` - Invalid input or configuration error
|
||||
|
||||
#### stella groundtruth bundle import
|
||||
|
||||
Import and verify evidence bundles in air-gapped environments.
|
||||
|
||||
```bash
|
||||
stella groundtruth bundle import [options]
|
||||
```
|
||||
|
||||
**Options:**
|
||||
| Option | Description | Required |
|
||||
|--------|-------------|----------|
|
||||
| `--input <path>` | Input bundle path | Yes |
|
||||
| `--verify-signature` | Verify bundle signatures | No |
|
||||
| `--trusted-keys <path>` | Path to trusted public keys | No |
|
||||
| `--trust-profile <path>` | Trust profile for verification | No |
|
||||
| `--output <path>` | Output verification report | No |
|
||||
| `--format <fmt>` | Report format: `markdown`, `json`, `html` | No |
|
||||
|
||||
**Example:**
|
||||
```bash
|
||||
stella groundtruth bundle import \
|
||||
--input symbol-bundle.tar.gz \
|
||||
--verify-signature \
|
||||
--trusted-keys /etc/stellaops/trusted-keys.pub \
|
||||
--trust-profile /etc/stellaops/trust-profiles/global.json \
|
||||
--output verification-report.md
|
||||
```
|
||||
|
||||
**Verification Steps:**
|
||||
1. Validate bundle manifest signature
|
||||
2. Verify all blob digests match manifest
|
||||
3. Validate DSSE envelope signatures against trusted keys
|
||||
4. Verify RFC 3161 timestamps against trusted TSA certificates
|
||||
5. Run IR matcher to confirm patched functions
|
||||
6. Verify SBOM canonical hash matches signed predicate
|
||||
7. Output verification report with KPI line items
|
||||
|
||||
**Exit Codes:**
|
||||
- `0` - All verifications passed
|
||||
- `1` - One or more verifications failed
|
||||
- `2` - Invalid input or configuration error
|
||||
|
||||
---
|
||||
|
||||
### stella groundtruth validate check
|
||||
|
||||
Check KPI regression against baseline thresholds.
|
||||
|
||||
**Sprint:** SPRINT_20260121_036_BinaryIndex_golden_corpus_bundle_verification
|
||||
|
||||
```bash
|
||||
stella groundtruth validate check [options]
|
||||
```
|
||||
|
||||
**Options:**
|
||||
| Option | Description | Default |
|
||||
|--------|-------------|---------|
|
||||
| `--results <path>` | Path to validation results JSON | (required) |
|
||||
| `--baseline <path>` | Path to baseline JSON | (required) |
|
||||
| `--precision-threshold <pp>` | Max precision drop (percentage points) | 0.01 |
|
||||
| `--recall-threshold <pp>` | Max recall drop (percentage points) | 0.01 |
|
||||
| `--fn-rate-threshold <pp>` | Max FN rate increase (percentage points) | 0.01 |
|
||||
| `--determinism-threshold <rate>` | Min determinism rate | 1.0 |
|
||||
| `--ttfrp-threshold <pct>` | Max TTFRP p95 increase (percentage) | 0.20 |
|
||||
| `--output <path>` | Output report path | stdout |
|
||||
| `--format <fmt>` | Report format: `markdown`, `json` | `markdown` |
|
||||
|
||||
**Example:**
|
||||
```bash
|
||||
stella groundtruth validate check \
|
||||
--results bench/results/20260122.json \
|
||||
--baseline bench/baselines/current.json \
|
||||
--precision-threshold 0.01 \
|
||||
--recall-threshold 0.01 \
|
||||
--fn-rate-threshold 0.01 \
|
||||
--determinism-threshold 1.0 \
|
||||
--output regression-report.md
|
||||
```
|
||||
|
||||
**Regression Gates:**
|
||||
| Metric | Threshold | Action |
|
||||
|--------|-----------|--------|
|
||||
| Precision | Drops > threshold | Fail |
|
||||
| Recall | Drops > threshold | Fail |
|
||||
| False-negative rate | Increases > threshold | Fail |
|
||||
| Deterministic replay | Drops below threshold | Fail |
|
||||
| TTFRP p95 | Increases > threshold | Warn |
|
||||
|
||||
**Exit Codes:**
|
||||
- `0` - All gates passed
|
||||
- `1` - One or more gates failed
|
||||
- `2` - Invalid input or configuration error
|
||||
|
||||
---
|
||||
|
||||
### stella groundtruth baseline
|
||||
|
||||
Manage KPI baselines for regression detection.
|
||||
|
||||
**Sprint:** SPRINT_20260121_036_BinaryIndex_golden_corpus_bundle_verification
|
||||
|
||||
**Usage:**
|
||||
```bash
|
||||
stella groundtruth baseline <command> [options]
|
||||
```
|
||||
|
||||
**Subcommands:**
|
||||
- `update` - Update baseline from validation results
|
||||
- `show` - Display baseline contents
|
||||
|
||||
#### stella groundtruth baseline update
|
||||
|
||||
Update baseline from validation results.
|
||||
|
||||
```bash
|
||||
stella groundtruth baseline update [options]
|
||||
```
|
||||
|
||||
**Options:**
|
||||
| Option | Description | Required |
|
||||
|--------|-------------|----------|
|
||||
| `--from-results <path>` | Path to validation results JSON | Yes |
|
||||
| `--output <path>` | Output baseline path | Yes |
|
||||
| `--description <text>` | Description for the baseline update | No |
|
||||
| `--source <commit>` | Source commit SHA for traceability | No |
|
||||
|
||||
**Example:**
|
||||
```bash
|
||||
stella groundtruth baseline update \
|
||||
--from-results bench/results/20260122.json \
|
||||
--output bench/baselines/current.json \
|
||||
--description "Post algorithm-v2.3 update" \
|
||||
--source "$(git rev-parse HEAD)"
|
||||
```
|
||||
|
||||
#### stella groundtruth baseline show
|
||||
|
||||
Display baseline contents.
|
||||
|
||||
```bash
|
||||
stella groundtruth baseline show --baseline <path> [--format table|json]
|
||||
```
|
||||
|
||||
**Options:**
|
||||
| Option | Description | Default |
|
||||
|--------|-------------|---------|
|
||||
| `--baseline <path>` | Path to baseline JSON | (required) |
|
||||
| `--format` | Output format: `table`, `json` | `table` |
|
||||
|
||||
**Output (table):**
|
||||
```
|
||||
Baseline ID: baseline-20260122120000
|
||||
Created: 2026-01-22T12:00:00Z
|
||||
Source: abc123def456
|
||||
Description: Post-semantic-diffing-v2 baseline
|
||||
|
||||
KPIs:
|
||||
Precision: 0.9500
|
||||
Recall: 0.9200
|
||||
False Negative Rate: 0.0800
|
||||
Determinism: 1.0000
|
||||
TTFRP p95: 150ms
|
||||
```
|
||||
|
||||
**See Also:** [Ground-Truth CLI Guide](../ground-truth-cli.md)
|
||||
|
||||
---
|
||||
## Reporting & Export Commands
|
||||
|
||||
### stella report
|
||||
|
||||
351
docs/modules/cli/guides/ground-truth-cli.md
Normal file
351
docs/modules/cli/guides/ground-truth-cli.md
Normal file
@@ -0,0 +1,351 @@
|
||||
# Ground-Truth Corpus CLI Guide
|
||||
|
||||
**Sprint:** SPRINT_20260121_035_BinaryIndex_golden_corpus_connectors_cli
|
||||
|
||||
## Overview
|
||||
|
||||
The `stella groundtruth` command group provides CLI access to the ground-truth corpus for patch-paired binary verification. This corpus enables precision validation of security advisories by maintaining symbol and binary pairs from upstream distribution sources.
|
||||
|
||||
## Use Cases
|
||||
|
||||
- **Security teams**: Validate patch presence in production binaries
|
||||
- **Compliance auditors**: Generate evidence bundles for air-gapped verification
|
||||
- **DevSecOps**: Integrate corpus validation into CI/CD pipelines
|
||||
- **Researchers**: Query symbol databases for vulnerability analysis
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Stella CLI installed and configured
|
||||
- Backend connectivity to Platform service (or offline bundle)
|
||||
- For sync operations: network access to upstream sources
|
||||
|
||||
## Command Structure
|
||||
|
||||
```
|
||||
stella groundtruth
|
||||
├── sources # Manage symbol source connectors
|
||||
│ ├── list # List available connectors
|
||||
│ ├── enable # Enable a connector
|
||||
│ ├── disable # Disable a connector
|
||||
│ └── sync # Sync from upstream
|
||||
├── symbols # Query symbols in corpus
|
||||
│ ├── lookup # Lookup by debug ID
|
||||
│ └── search # Search by package/distro
|
||||
├── pairs # Manage security pairs
|
||||
│ ├── create # Create vuln/patch pair
|
||||
│ ├── list # List existing pairs
|
||||
│ └── delete # Remove a pair
|
||||
└── validate # Run validation harness
|
||||
├── run # Execute validation
|
||||
├── metrics # View run metrics
|
||||
└── export # Export report
|
||||
```
|
||||
|
||||
## Source Connectors
|
||||
|
||||
The ground-truth corpus ingests data from multiple upstream sources:
|
||||
|
||||
| Connector ID | Distribution | Data Type | Description |
|
||||
|--------------|--------------|-----------|-------------|
|
||||
| `debuginfod-fedora` | Fedora | Debug symbols | ELF debuginfo via debuginfod protocol |
|
||||
| `debuginfod-ubuntu` | Ubuntu | Debug symbols | ELF debuginfo via debuginfod protocol |
|
||||
| `ddeb-ubuntu` | Ubuntu | Debug packages | `.ddeb` debug symbol packages |
|
||||
| `buildinfo-debian` | Debian | Build metadata | `.buildinfo` reproducibility records |
|
||||
| `secdb-alpine` | Alpine | Security DB | `secfixes` YAML from APKBUILD |
|
||||
|
||||
### List Sources
|
||||
|
||||
```bash
|
||||
stella groundtruth sources list
|
||||
|
||||
# Output:
|
||||
ID Display Name Status Last Sync
|
||||
------------------------------------------------------------------------------------------
|
||||
debuginfod-fedora Fedora Debuginfod Enabled 2026-01-22T10:00:00Z
|
||||
debuginfod-ubuntu Ubuntu Debuginfod Enabled 2026-01-22T10:00:00Z
|
||||
ddeb-ubuntu Ubuntu ddebs Enabled 2026-01-22T09:30:00Z
|
||||
buildinfo-debian Debian Buildinfo Enabled 2026-01-22T08:00:00Z
|
||||
secdb-alpine Alpine SecDB Enabled 2026-01-22T06:00:00Z
|
||||
```
|
||||
|
||||
### Enable/Disable Sources
|
||||
|
||||
```bash
|
||||
# Enable a source connector
|
||||
stella groundtruth sources enable debuginfod-fedora
|
||||
|
||||
# Disable a source connector (stops future syncs)
|
||||
stella groundtruth sources disable debuginfod-fedora
|
||||
```
|
||||
|
||||
### Sync Sources
|
||||
|
||||
```bash
|
||||
# Incremental sync of all enabled sources
|
||||
stella groundtruth sources sync
|
||||
|
||||
# Full sync of a specific source
|
||||
stella groundtruth sources sync --source buildinfo-debian --full
|
||||
|
||||
# Sync with verbose output
|
||||
stella groundtruth sources sync --source ddeb-ubuntu -v
|
||||
```
|
||||
|
||||
## Symbol Operations
|
||||
|
||||
### Lookup by Debug ID
|
||||
|
||||
Query symbols using the ELF GNU Build-ID or equivalent identifier:
|
||||
|
||||
```bash
|
||||
# Lookup by build-id
|
||||
stella groundtruth symbols lookup --debug-id 7f8a9b2c4d5e6f1a
|
||||
|
||||
# JSON output
|
||||
stella groundtruth symbols lookup --debug-id 7f8a9b2c4d5e6f1a --output-format json
|
||||
```
|
||||
|
||||
**Example output:**
|
||||
```
|
||||
Binary: libcrypto.so.3
|
||||
Architecture: x86_64
|
||||
Distribution: debian-bookworm
|
||||
Package: openssl@3.0.11-1
|
||||
Symbol Count: 4523
|
||||
Sources: debuginfod-fedora, buildinfo-debian
|
||||
```
|
||||
|
||||
### Search Symbols
|
||||
|
||||
Search across the corpus by package name or distribution:
|
||||
|
||||
```bash
|
||||
# Search by package
|
||||
stella groundtruth symbols search --package openssl
|
||||
|
||||
# Filter by distribution
|
||||
stella groundtruth symbols search --package openssl --distro debian
|
||||
|
||||
# Limit results
|
||||
stella groundtruth symbols search --package curl --limit 100
|
||||
```
|
||||
|
||||
## Security Pairs
|
||||
|
||||
Security pairs link vulnerable and patched binary versions for a specific CVE.
|
||||
|
||||
### Create a Pair
|
||||
|
||||
```bash
|
||||
stella groundtruth pairs create \
|
||||
--cve CVE-2024-1234 \
|
||||
--vuln-pkg openssl=3.0.10-1 \
|
||||
--patch-pkg openssl=3.0.11-1 \
|
||||
--distro debian-bookworm
|
||||
```
|
||||
|
||||
### List Pairs
|
||||
|
||||
```bash
|
||||
# List all pairs
|
||||
stella groundtruth pairs list
|
||||
|
||||
# Filter by CVE pattern
|
||||
stella groundtruth pairs list --cve "CVE-2024-*"
|
||||
|
||||
# Filter by package
|
||||
stella groundtruth pairs list --package openssl --limit 50
|
||||
|
||||
# JSON output
|
||||
stella groundtruth pairs list --output-format json
|
||||
```
|
||||
|
||||
**Example output:**
|
||||
```
|
||||
Pair ID CVE Package Vuln Version Patch Version
|
||||
-------------------------------------------------------------------------------
|
||||
pair-001 CVE-2024-1234 openssl 3.0.10-1 3.0.11-1
|
||||
pair-002 CVE-2024-5678 curl 8.4.0-1 8.5.0-1
|
||||
```
|
||||
|
||||
### Delete a Pair
|
||||
|
||||
```bash
|
||||
# Delete with confirmation prompt
|
||||
stella groundtruth pairs delete pair-001
|
||||
|
||||
# Skip confirmation
|
||||
stella groundtruth pairs delete pair-001 --force
|
||||
```
|
||||
|
||||
## Validation Harness
|
||||
|
||||
The validation harness runs end-to-end verification against security pairs.
|
||||
|
||||
### Run Validation
|
||||
|
||||
```bash
|
||||
# Validate all pairs
|
||||
stella groundtruth validate run
|
||||
|
||||
# Validate specific pairs (pattern match)
|
||||
stella groundtruth validate run --pairs "openssl:CVE-2024-*"
|
||||
|
||||
# Use specific matcher
|
||||
stella groundtruth validate run --matcher semantic-diffing
|
||||
|
||||
# Parallel validation with report output
|
||||
stella groundtruth validate run \
|
||||
--pairs "curl:*" \
|
||||
--parallel 8 \
|
||||
--output validation-report.md
|
||||
```
|
||||
|
||||
**Matcher types:**
|
||||
| Matcher | Description |
|
||||
|---------|-------------|
|
||||
| `semantic-diffing` | IR-level semantic comparison (default) |
|
||||
| `hash-based` | Function hash matching |
|
||||
| `hybrid` | Combined semantic + hash approach |
|
||||
|
||||
### View Metrics
|
||||
|
||||
```bash
|
||||
stella groundtruth validate metrics --run-id vr-20260122100532
|
||||
|
||||
# JSON output
|
||||
stella groundtruth validate metrics --run-id vr-20260122100532 --output-format json
|
||||
```
|
||||
|
||||
**Example output:**
|
||||
```
|
||||
Run ID: vr-20260122100532
|
||||
Duration: 2026-01-22T10:00:00Z - 2026-01-22T10:15:32Z
|
||||
Pairs: 48/50 successful
|
||||
Function Match Rate: 94.2%
|
||||
False-Negative Rate: 2.1%
|
||||
SBOM Hash Stability: 3/3
|
||||
Verify Time (p50/p95): 423ms / 1.2s
|
||||
```
|
||||
|
||||
### Export Reports
|
||||
|
||||
```bash
|
||||
# Export as Markdown
|
||||
stella groundtruth validate export \
|
||||
--run-id vr-20260122100532 \
|
||||
--format markdown \
|
||||
--output report.md
|
||||
|
||||
# Export as HTML
|
||||
stella groundtruth validate export \
|
||||
--run-id vr-20260122100532 \
|
||||
--format html \
|
||||
--output report.html
|
||||
|
||||
# Export as JSON (machine-readable)
|
||||
stella groundtruth validate export \
|
||||
--run-id vr-20260122100532 \
|
||||
--format json \
|
||||
--output report.json
|
||||
```
|
||||
|
||||
## CI/CD Integration
|
||||
|
||||
### GitHub Actions Example
|
||||
|
||||
```yaml
|
||||
name: Corpus Validation
|
||||
on:
|
||||
schedule:
|
||||
- cron: '0 6 * * 1' # Weekly on Monday
|
||||
|
||||
jobs:
|
||||
validate:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- name: Sync corpus sources
|
||||
run: stella groundtruth sources sync
|
||||
|
||||
- name: Run validation
|
||||
run: |
|
||||
stella groundtruth validate run \
|
||||
--matcher semantic-diffing \
|
||||
--parallel 4 \
|
||||
--output validation-${{ github.run_id }}.md
|
||||
|
||||
- name: Check metrics
|
||||
run: |
|
||||
MATCH_RATE=$(stella groundtruth validate metrics --run-id $(cat run-id.txt) --output-format json | jq '.functionMatchRate')
|
||||
if (( $(echo "$MATCH_RATE < 90" | bc -l) )); then
|
||||
echo "Match rate below threshold: $MATCH_RATE%"
|
||||
exit 1
|
||||
fi
|
||||
```
|
||||
|
||||
### GitLab CI Example
|
||||
|
||||
```yaml
|
||||
corpus-validation:
|
||||
stage: verify
|
||||
script:
|
||||
- stella groundtruth sources sync --source buildinfo-debian
|
||||
- stella groundtruth validate run --pairs "openssl:*" --output report.md
|
||||
artifacts:
|
||||
paths:
|
||||
- report.md
|
||||
expire_in: 1 week
|
||||
rules:
|
||||
- if: $CI_PIPELINE_SOURCE == "schedule"
|
||||
```
|
||||
|
||||
## Offline Usage
|
||||
|
||||
For air-gapped environments, use offline bundles:
|
||||
|
||||
```bash
|
||||
# Export corpus for offline use
|
||||
stella bundle export \
|
||||
--include-corpus \
|
||||
--output corpus-bundle-$(date +%F).tar.gz
|
||||
|
||||
# Import on air-gapped system
|
||||
stella bundle import --package corpus-bundle-2026-01-22.tar.gz
|
||||
|
||||
# Run validation offline
|
||||
stella groundtruth validate run --offline
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Common Issues
|
||||
|
||||
**Sync fails with network error:**
|
||||
```bash
|
||||
# Check source status
|
||||
stella groundtruth sources list
|
||||
|
||||
# Retry with verbose output
|
||||
stella groundtruth sources sync --source debuginfod-ubuntu -v
|
||||
```
|
||||
|
||||
**Symbol lookup returns no results:**
|
||||
```bash
|
||||
# Verify debug-id format (hex string)
|
||||
stella groundtruth symbols lookup --debug-id abc123 -v
|
||||
|
||||
# Try searching by package instead
|
||||
stella groundtruth symbols search --package libcrypto
|
||||
```
|
||||
|
||||
**Validation metrics show low match rate:**
|
||||
- Check that both vuln and patch binaries are present in corpus
|
||||
- Verify symbol sources are synced and enabled
|
||||
- Consider using `hybrid` matcher for complex cases
|
||||
|
||||
## See Also
|
||||
|
||||
- [CLI Command Reference](commands/reference.md#ground-truth-corpus-commands)
|
||||
- [BinaryIndex Architecture](../../binary-index/architecture.md)
|
||||
- [Golden Corpus KPIs](../../benchmarks/golden-corpus-kpis.md)
|
||||
- [Air-Gap Bundle Guide](../../modules/airgap/README.md)
|
||||
36
docs/modules/cli/guides/trust-profiles.md
Normal file
36
docs/modules/cli/guides/trust-profiles.md
Normal file
@@ -0,0 +1,36 @@
|
||||
# Trust Profiles
|
||||
|
||||
Trust profiles are offline trust-store templates for bundle verification. They define trust roots, Rekor public keys, and TSA roots in a single file so operators can apply a profile into a local trust store.
|
||||
|
||||
Default profile location:
|
||||
- `etc/trust-profiles/*.trustprofile.json`
|
||||
- Assets referenced by profiles live under `etc/trust-profiles/assets/`
|
||||
|
||||
Profile structure (summary):
|
||||
- `profileId`: stable identifier (used by CLI commands)
|
||||
- `trustRoots[]`: signing trust roots (PEM files)
|
||||
- `rekorKeys[]`: Rekor public keys for offline inclusion proof verification
|
||||
- `tsaRoots[]`: TSA roots for RFC3161 verification
|
||||
- `metadata`: optional compliance metadata
|
||||
|
||||
CLI usage:
|
||||
- `stella trust-profile list`
|
||||
- `stella trust-profile show <profile-id>`
|
||||
- `stella trust-profile apply <profile-id> --output <dir>`
|
||||
|
||||
Profile lookup overrides:
|
||||
- `--profiles-dir <path>` to point at a custom profiles directory
|
||||
- `STELLAOPS_TRUST_PROFILES` environment variable for default lookup
|
||||
|
||||
Apply output:
|
||||
- `trust-manifest.json` (trust roots manifest for offline verification)
|
||||
- `trust-profile.json` (resolved profile copy)
|
||||
- `trust-root.pem` (combined trust roots for CLI verification)
|
||||
- `trust-roots/`, `rekor/`, `tsa/` folders with PEM assets
|
||||
|
||||
Example apply workflow:
|
||||
1. `stella trust-profile apply global --output ./trust-store`
|
||||
2. `stella bundle verify --trust-root ./trust-store/trust-root.pem`
|
||||
|
||||
Note:
|
||||
- Default profiles ship with placeholder roots for scaffolding only. Replace them with compliance-approved roots before production use.
|
||||
Reference in New Issue
Block a user