stella-ops.org/git.stella-ops.org
1
0
Fork 0
Code Issues Pull Requests Actions 2 Releases Wiki Activity

synergy moats product advisory implementations

Browse Source
This commit is contained in:
master
2026-01-17 01:30:03 +02:00
parent 77ff029205
commit 702a27ac83
112 changed files with 21356 additions and 127 deletions
Download Patch File Download Diff File Expand all files Collapse all files

271
docs/modules/cli/guides/audit-bundle-format.md Normal file
View File

@@ -0,0 +1,271 @@
# Audit Bundle Format Specification
> **Sprint:** SPRINT_20260117_027_CLI_audit_bundle_command
> **Task:** AUD-001 - Audit Bundle Format Specification
> **Version:** 1.0.0
## Overview
The Stella Ops Audit Bundle is a self-contained, tamper-evident package containing all evidence required for an auditor to verify a release decision. The bundle is designed for:
- **Completeness:** Contains everything needed to verify a verdict without additional tool invocations
- **Reproducibility:** Includes replay instructions for deterministic re-verification
- **Portability:** Standard formats (JSON, Markdown) readable by common tools
- **Integrity:** Cryptographic manifest ensures tamper detection
## Bundle Structure
```
audit-bundle-<digest>-<timestamp>/
├── manifest.json # Bundle manifest with cryptographic hashes
├── README.md # Human-readable guide for auditors
├── verdict/
│ ├── verdict.json # StellaVerdict artifact
│ └── verdict.dsse.json # DSSE envelope with signatures
├── evidence/
│ ├── sbom.json # SBOM (CycloneDX format)
│ ├── vex-statements/ # All VEX statements considered
│ │ ├── index.json # VEX index with sources
│ │ └── *.json # Individual VEX documents
│ ├── reachability/
│ │ ├── analysis.json # Reachability analysis result
│ │ └── call-graph.dot # Call graph visualization (optional)
│ └── provenance/
│ └── slsa-provenance.json
├── policy/
│ ├── policy-snapshot.json # Policy version and rules used
│ ├── gate-decision.json # Gate evaluation result
│ └── evaluation-trace.json # Full policy trace (optional)
├── replay/
│ ├── knowledge-snapshot.json # Frozen inputs for replay
│ └── replay-instructions.md # How to replay verdict
└── schema/ # Schema references (optional)
├── verdict-schema.json
└── vex-schema.json
```
## File Specifications
### manifest.json
The manifest provides cryptographic integrity and bundle metadata.
```json
{
"$schema": "https://schema.stella-ops.org/audit-bundle/manifest/v1",
"version": "1.0.0",
"bundleId": "urn:stella:audit-bundle:sha256:abc123...",
"artifactDigest": "sha256:abc123...",
"generatedAt": "2026-01-17T10:30:00Z",
"generatedBy": "stella-cli/2.5.0",
"files": [
{
"path": "verdict/verdict.json",
"sha256": "abc123...",
"size": 12345,
"required": true
},
{
"path": "evidence/sbom.json",
"sha256": "def456...",
"size": 98765,
"required": true
}
],
"totalFiles": 12,
"totalSize": 234567,
"integrityHash": "sha256:manifest-hash-of-all-file-hashes"
}
```
### README.md
Auto-generated guide for auditors with:
- Bundle overview and artifact identification
- Quick verification steps
- File inventory with descriptions
- Contact information for questions
### verdict/verdict.json
The StellaVerdict artifact in standard format:
```json
{
"$schema": "https://schema.stella-ops.org/verdict/v1",
"artifactDigest": "sha256:abc123...",
"artifactType": "container-image",
"decision": "BLOCKED",
"timestamp": "2026-01-17T10:25:00Z",
"gates": [
{
"gateId": "vex-trust",
"status": "BLOCKED",
"reason": "Trust score below threshold (0.45 < 0.70)",
"evidenceRefs": ["evidence/vex-statements/vendor-x.json"]
}
],
"contentId": "urn:stella:verdict:sha256:xyz..."
}
```
### verdict/verdict.dsse.json
DSSE (Dead Simple Signing Envelope) containing the signed verdict:
```json
{
"payloadType": "application/vnd.stella-ops.verdict+json",
"payload": "base64-encoded-verdict",
"signatures": [
{
"keyid": "urn:stella:key:sha256:...",
"sig": "base64-signature"
}
]
}
```
### evidence/sbom.json
CycloneDX SBOM in JSON format (or SPDX if configured).
### evidence/vex-statements/
Directory containing all VEX statements considered during evaluation:
- `index.json` - Index of VEX statements with metadata
- Individual VEX documents named by source and ID
### evidence/reachability/analysis.json
Reachability analysis results:
```json
{
"artifactDigest": "sha256:abc123...",
"analysisType": "static",
"analysisTimestamp": "2026-01-17T10:20:00Z",
"components": [
{
"purl": "pkg:npm/lodash@4.17.21",
"vulnerabilities": [
{
"id": "CVE-2021-23337",
"reachable": false,
"reason": "Vulnerable function not in call graph"
}
]
}
]
}
```
### policy/policy-snapshot.json
Snapshot of policy configuration at evaluation time:
```json
{
"policyVersion": "v2.3.1",
"policyDigest": "sha256:policy-hash...",
"gates": ["sbom-required", "vex-trust", "cve-threshold"],
"thresholds": {
"vexTrustScore": 0.70,
"maxCriticalCves": 0,
"maxHighCves": 5
},
"evaluatedAt": "2026-01-17T10:25:00Z"
}
```
### policy/gate-decision.json
Detailed gate evaluation result:
```json
{
"artifactDigest": "sha256:abc123...",
"overallDecision": "BLOCKED",
"gates": [
{
"gateId": "vex-trust",
"decision": "BLOCKED",
"inputs": {
"vexStatements": 3,
"trustScore": 0.45,
"threshold": 0.70
},
"reason": "Trust score below threshold",
"suggestion": "Obtain VEX from trusted issuer or adjust trust registry"
}
]
}
```
### replay/knowledge-snapshot.json
Frozen inputs for deterministic replay:
```json
{
"$schema": "https://schema.stella-ops.org/knowledge-snapshot/v1",
"snapshotId": "urn:stella:snapshot:sha256:...",
"capturedAt": "2026-01-17T10:25:00Z",
"inputs": {
"sbomDigest": "sha256:sbom-hash...",
"vexStatements": ["sha256:vex1...", "sha256:vex2..."],
"policyDigest": "sha256:policy-hash...",
"reachabilityDigest": "sha256:reach-hash..."
},
"replayCommand": "stella replay snapshot --manifest replay/knowledge-snapshot.json"
}
```
### replay/replay-instructions.md
Human-readable replay instructions (auto-generated, see AUD-004).
## Archive Formats
The bundle can be output in three formats:
| Format | Extension | Use Case |
|--------|-----------|----------|
| Directory | (none) | Local inspection, development |
| tar.gz | `.tar.gz` | Transfer, archival (default for remote) |
| zip | `.zip` | Windows compatibility |
## Verification
To verify a bundle's integrity:
```bash
stella audit verify ./audit-bundle-sha256-abc123/
```
Verification checks:
1. Parse `manifest.json`
2. Verify each file's SHA-256 hash matches manifest
3. Verify `integrityHash` (hash of all file hashes)
4. Optionally verify DSSE signatures
## Compliance Mapping
| Compliance Framework | Bundle Component |
|---------------------|------------------|
| SOC 2 (CC7.1) | verdict/, policy/ |
| ISO 27001 (A.12.6) | evidence/sbom.json |
| FedRAMP | All components |
| SLSA Level 3 | evidence/provenance/ |
## Extensibility
Custom evidence can be added to `evidence/custom/` directory. Custom files must be:
- Listed in `manifest.json`
- JSON or Markdown format
- Include schema reference if JSON
---
_Last updated: 2026-01-17 (UTC)_

251
docs/modules/cli/guides/commands/audit.md Normal file
View File

@@ -0,0 +1,251 @@
# stella audit
> **Sprint:** SPRINT_20260117_027_CLI_audit_bundle_command
> **Task:** AUD-007 - Documentation
Commands for audit operations including bundle generation and verification.
## Synopsis
```
stella audit <command> [options]
```
## Commands
| Command | Description |
|---------|-------------|
| `bundle` | Generate self-contained audit bundle for an artifact |
| `verify` | Verify audit bundle integrity |
---
## stella audit bundle
Generate a self-contained, auditor-ready evidence package for an artifact.
### Synopsis
```
stella audit bundle <digest> [options]
```
### Arguments
| Argument | Description |
|----------|-------------|
| `<digest>` | Artifact digest (e.g., `sha256:abc123...`) |
### Options
| Option | Default | Description |
|--------|---------|-------------|
| `--output <path>` | `./audit-bundle-<digest>/` | Output path for the bundle |
| `--format <format>` | `dir` | Output format: `dir`, `tar.gz`, `zip` |
| `--include-call-graph` | `false` | Include call graph visualization |
| `--include-schemas` | `false` | Include JSON schema files |
| `--include-trace` | `true` | Include policy evaluation trace |
| `--policy-version <ver>` | (current) | Use specific policy version |
| `--overwrite` | `false` | Overwrite existing output |
| `--verbose` | `false` | Show progress during generation |
### Examples
```bash
# Generate bundle as directory
stella audit bundle sha256:abc123def456
# Generate tar.gz archive
stella audit bundle sha256:abc123def456 --format tar.gz
# Specify output location
stella audit bundle sha256:abc123def456 --output ./audits/release-v2.5/
# Include all optional content
stella audit bundle sha256:abc123def456 \
--include-call-graph \
--include-schemas \
--verbose
# Use specific policy version
stella audit bundle sha256:abc123def456 --policy-version v2.3.1
```
### Output
The bundle contains:
```
audit-bundle-<digest>-<timestamp>/
├── manifest.json # Bundle manifest with cryptographic hashes
├── README.md # Human-readable guide for auditors
├── verdict/
│ ├── verdict.json # StellaVerdict artifact
│ └── verdict.dsse.json # DSSE envelope with signatures
├── evidence/
│ ├── sbom.json # SBOM (CycloneDX format)
│ ├── vex-statements/ # All VEX statements considered
│ │ ├── index.json
│ │ └── *.json
│ ├── reachability/
│ │ ├── analysis.json
│ │ └── call-graph.dot # Optional
│ └── provenance/
│ └── slsa-provenance.json
├── policy/
│ ├── policy-snapshot.json
│ ├── gate-decision.json
│ └── evaluation-trace.json
├── replay/
│ ├── knowledge-snapshot.json
│ └── replay-instructions.md
└── schema/ # Optional
├── verdict-schema.json
└── vex-schema.json
```
### Exit Codes
| Code | Description |
|------|-------------|
| 0 | Bundle generated successfully |
| 1 | Bundle generated with missing evidence (warnings) |
| 2 | Error (artifact not found, permission denied, etc.) |
---
## stella audit verify
Verify the integrity of an audit bundle.
### Synopsis
```
stella audit verify <bundle-path> [options]
```
### Arguments
| Argument | Description |
|----------|-------------|
| `<bundle-path>` | Path to audit bundle (directory or archive) |
### Options
| Option | Default | Description |
|--------|---------|-------------|
| `--strict` | `false` | Fail on any missing optional files |
| `--check-signatures` | `false` | Verify DSSE signatures |
| `--trusted-keys <path>` | (none) | Path to trusted keys file for signature verification |
### Examples
```bash
# Basic verification
stella audit verify ./audit-bundle-abc123-20260117/
# Strict mode (fail on any missing files)
stella audit verify ./audit-bundle-abc123-20260117/ --strict
# Verify signatures
stella audit verify ./audit-bundle.tar.gz \
--check-signatures \
--trusted-keys ./trusted-keys.json
# Verify archive directly
stella audit verify ./audit-bundle-abc123.zip
```
### Output
```
Verifying bundle: ./audit-bundle-abc123-20260117/
Bundle ID: urn:stella:audit-bundle:sha256:abc123...
Artifact: sha256:abc123def456...
Generated: 2026-01-17T10:30:00Z
Files: 15
Verifying files...
✓ Verified 15/15 files
✓ Integrity hash verified
✓ Bundle integrity verified
```
### Exit Codes
| Code | Description |
|------|-------------|
| 0 | Bundle is valid |
| 1 | Bundle integrity check failed |
| 2 | Error (bundle not found, invalid format, etc.) |
---
## Trusted Keys File Format
For signature verification, provide a JSON file with trusted public keys:
```json
{
"keys": [
{
"keyId": "urn:stella:key:sha256:abc123...",
"publicKey": "-----BEGIN PUBLIC KEY-----\n...\n-----END PUBLIC KEY-----"
}
]
}
```
---
## Use Cases
### Generating Bundles for External Auditors
```bash
# Generate comprehensive bundle for SOC 2 audit
stella audit bundle sha256:prod-release-v2.5 \
--format zip \
--include-schemas \
--output ./soc2-audit-2026/release-evidence.zip
```
### Verifying Received Bundles
```bash
# Verify bundle received from another team
stella audit verify ./received-bundle.tar.gz --strict
# Verify with signature checking
stella audit verify ./received-bundle/ \
--check-signatures \
--trusted-keys ./company-signing-keys.json
```
### CI/CD Integration
```yaml
# GitLab CI example
audit-bundle:
stage: release
script:
- stella audit bundle $IMAGE_DIGEST --format tar.gz --output ./audit/
artifacts:
paths:
- audit/
expire_in: 5 years
```
---
## Related
- [Audit Bundle Format Specification](audit-bundle-format.md)
- [stella replay](../replay.md) - Replay verdicts for verification
- [stella export](export.md) - Export evidence in various formats
---
_Last updated: 2026-01-17 (UTC)_

313
docs/modules/cli/guides/commands/explain.md Normal file
View File

@@ -0,0 +1,313 @@
# stella explain - Block Explanation Commands
**Sprint:** SPRINT_20260117_026_CLI_why_blocked_command
## Overview
The `stella explain` command group provides commands for understanding why artifacts are blocked by policy gates. This addresses the M2 moat requirement: **"Explainability with proof, not narrative."**
When an artifact is blocked, `stella explain` produces a **deterministic trace** with **referenced evidence artifacts**, enabling:
- Clear understanding of which gate blocked the artifact
- Actionable suggestions for remediation
- Verifiable evidence chain
- Deterministic replay for verification
---
## Commands
### stella explain block
Explain why an artifact was blocked by policy gates.
**Usage:**
```bash
stella explain block <digest> [options]
```
**Arguments:**
- `<digest>` - Artifact digest in any of these formats:
- `sha256:abc123...` - Full digest with algorithm prefix
- `abc123...` - Raw 64-character hex digest (assumed sha256)
- `registry.example.com/image@sha256:abc123...` - OCI reference (digest extracted)
**Options:**
| Option | Alias | Description | Default |
|--------|-------|-------------|---------|
| `--format <format>` | `-f` | Output format: `table`, `json`, `markdown` | `table` |
| `--show-evidence` | `-e` | Include full evidence artifact details | false |
| `--show-trace` | `-t` | Include policy evaluation trace | false |
| `--replay-token` | `-r` | Include replay token in output | false |
| `--output <path>` | `-o` | Write to file instead of stdout | stdout |
| `--offline` | | Query local verdict cache only | false |
---
## Output Formats
### Table Format (Default)
Human-readable format optimized for terminal display:
```
Artifact: sha256:abc123def456789012345678901234567890123456789012345678901234
Status: BLOCKED
Gate: VexTrust
Reason: Trust score below threshold (0.45 < 0.70)
Suggestion: Obtain VEX statement from trusted issuer or add issuer to trust registry
Evidence:
[VEX ] vex:sha256:de...23 vendor-x 2026-01-15T10:00:00Z
[REACH ] reach:sha256...56 static 2026-01-15T09:55:00Z
Replay: stella verify verdict --verdict urn:stella:verdict:sha256:abc123:v2.3.0:1737108000
```
### JSON Format
Machine-readable format for CI/CD integration:
```json
{
"artifact": "sha256:abc123def456789012345678901234567890123456789012345678901234",
"status": "BLOCKED",
"gate": "VexTrust",
"reason": "Trust score below threshold (0.45 < 0.70)",
"suggestion": "Obtain VEX statement from trusted issuer or add issuer to trust registry",
"evaluationTime": "2026-01-15T10:30:00+00:00",
"policyVersion": "v2.3.0",
"evidence": [
{
"type": "VEX",
"id": "vex:sha256:def456789abc123",
"source": "vendor-x",
"timestamp": "2026-01-15T10:00:00+00:00",
"retrieveCommand": "stella evidence get vex:sha256:def456789abc123"
},
{
"type": "REACH",
"id": "reach:sha256:789abc123def456",
"source": "static-analysis",
"timestamp": "2026-01-15T09:55:00+00:00",
"retrieveCommand": "stella evidence get reach:sha256:789abc123def456"
}
],
"replayCommand": "stella verify verdict --verdict urn:stella:verdict:sha256:abc123:v2.3.0:1737108000"
}
```
### Markdown Format
Suitable for embedding in GitHub issues, PR comments, or documentation:
```markdown
## Block Explanation
**Artifact:** `sha256:abc123def456789012345678901234567890123456789012345678901234`
**Status:** BLOCKED
### Gate Decision
| Property | Value |
|----------|-------|
| Gate | VexTrust |
| Reason | Trust score below threshold (0.45 < 0.70) |
| Suggestion | Obtain VEX statement from trusted issuer or add issuer to trust registry |
| Policy Version | v2.3.0 |
### Evidence
| Type | ID | Source | Timestamp |
|------|-----|--------|-----------|
| VEX | `vex:sha256:de...23` | vendor-x | 2026-01-15 10:00 |
| REACH | `reach:sha256...56` | static-analysis | 2026-01-15 09:55 |
### Verification
```bash
stella verify verdict --verdict urn:stella:verdict:sha256:abc123:v2.3.0:1737108000
```
```
---
## Examples
### Basic Block Explanation
```bash
# Get basic explanation of why an artifact is blocked
stella explain block sha256:abc123def456789012345678901234567890123456789012345678901234
```
### JSON Output for CI/CD
```bash
# Get JSON output for parsing in CI/CD pipeline
stella explain block sha256:abc123... --format json --output block-reason.json
# Parse in CI/CD
GATE=$(jq -r '.gate' block-reason.json)
REASON=$(jq -r '.reason' block-reason.json)
echo "Blocked by $GATE: $REASON"
```
### Full Explanation with Evidence and Trace
```bash
# Get complete explanation with all details
stella explain block sha256:abc123... \
--show-evidence \
--show-trace \
--replay-token \
--format table
```
### Markdown for PR Comment
```bash
# Generate markdown for GitHub PR comment
stella explain block sha256:abc123... --format markdown --output comment.md
# Use with gh CLI
gh pr comment 123 --body-file comment.md
```
### Retrieve Evidence Artifacts
```bash
# Get explanation
stella explain block sha256:abc123... --show-evidence
# Retrieve specific evidence artifacts
stella evidence get vex:sha256:def456789abc123
stella evidence get reach:sha256:789abc123def456
```
### Verify Deterministic Replay
```bash
# Get replay token
REPLAY=$(stella explain block sha256:abc123... --format json | jq -r '.replayCommand')
# Execute replay verification
eval $REPLAY
```
---
## Exit Codes
| Code | Meaning |
|------|---------|
| `0` | Artifact is NOT blocked (all gates passed) |
| `1` | Artifact IS blocked (one or more gates failed) |
| `2` | Error (artifact not found, API error, etc.) |
**CI/CD Integration:**
```bash
# Fail pipeline if artifact is blocked
if ! stella explain block sha256:abc123... --format json > /dev/null 2>&1; then
EXIT_CODE=$?
if [ $EXIT_CODE -eq 1 ]; then
echo "ERROR: Artifact is blocked by policy"
stella explain block sha256:abc123... --format markdown
exit 1
else
echo "ERROR: Could not retrieve block status"
exit 2
fi
fi
```
---
## Evidence Types
The `explain block` command returns evidence artifacts that contributed to the gate decision:
| Type | Description | Source |
|------|-------------|--------|
| `VEX` | VEX (Vulnerability Exploitability eXchange) statement | VEX issuers, vendor security teams |
| `REACH` | Reachability analysis result | Static analysis, call graph analysis |
| `SBOM` | Software Bill of Materials | SBOM generators, build systems |
| `SCAN` | Vulnerability scan result | Scanner service |
| `ATTEST` | Attestation document | Attestor service, SLSA provenance |
| `POLICY` | Policy evaluation result | Policy engine |
---
## Determinism Guarantee
All output from `stella explain block` is **deterministic**:
1. **Same inputs produce identical outputs** - Given the same artifact digest and policy version, the output is byte-for-byte identical
2. **Evidence is sorted** - Evidence artifacts are sorted by timestamp (ascending)
3. **Trace is sorted** - Evaluation trace steps are sorted by step number
4. **Timestamps use ISO 8601** - All timestamps use ISO 8601 format with UTC offset
5. **JSON uses canonical ordering** - JSON properties are ordered consistently
This enables:
- **Replay verification** - Use the replay token to verify the decision can be reproduced
- **Audit trails** - Compare explanations across time
- **Cache validation** - Verify cached decisions match current evaluation
---
## Troubleshooting
### Artifact Not Found
```
Error: Artifact sha256:abc123... not found in registry or evidence store.
```
**Causes:**
- Artifact was never scanned
- Artifact digest is incorrect
- Artifact was deleted from registry
**Solutions:**
```bash
# Verify artifact exists
stella image inspect sha256:abc123...
# Scan the artifact
stella scan docker://myregistry/myimage@sha256:abc123...
```
### Not Blocked
```
Artifact sha256:abc123... is NOT blocked. All policy gates passed.
```
This means the artifact passed all policy evaluations. Exit code will be `0`.
### API Error
```
Error: Policy service unavailable
```
**Solutions:**
```bash
# Check connectivity
stella doctor --check check.policy.connectivity
# Use offline mode if available
stella explain block sha256:abc123... --offline
```
---
## See Also
- [Policy Commands](policy.md) - Policy management and testing
- [VEX Commands](vex.md) - VEX document management
- [Evidence Commands](evidence.md) - Evidence retrieval and verification
- [Verify Commands](verify.md) - Verdict verification and replay
- [Command Reference](reference.md) - Complete command reference

68
docs/modules/cli/guides/commands/reference.md
View File

@@ -13,6 +13,7 @@ graph TD
CLI --> ADMIN[Administration]
CLI --> AUTH[Authentication]
CLI --> POLICY[Policy Management]
CLI --> EXPLAIN[Explainability]
CLI --> VEX[VEX & Decisioning]
CLI --> SBOM[SBOM Operations]
CLI --> REPORT[Reporting & Export]
@@ -914,6 +915,73 @@ Platform: linux-x64
---
## Explainability Commands
### stella explain block
Explain why an artifact was blocked by policy gates. Produces deterministic trace with referenced evidence artifacts.
**Sprint:** SPRINT_20260117_026_CLI_why_blocked_command
**Moat Reference:** M2 (Explainability with proof, not narrative)
**Usage:**
```bash
stella explain block <digest> [options]
```
**Arguments:**
- `<digest>` - Artifact digest (`sha256:abc123...`, raw hex, or OCI reference)
**Options:**
| Option | Description | Default |
|--------|-------------|---------|
| `--format <format>` | Output format: `table`, `json`, `markdown` | `table` |
| `--show-evidence` | Include full evidence artifact details | false |
| `--show-trace` | Include policy evaluation trace | false |
| `--replay-token` | Include replay token in output | false |
| `--output <path>` | Write to file instead of stdout | stdout |
| `--offline` | Query local verdict cache only | false |
**Examples:**
```bash
# Basic explanation
stella explain block sha256:abc123def456...
# JSON output for CI/CD
stella explain block sha256:abc123... --format json --output reason.json
# Full explanation with evidence and trace
stella explain block sha256:abc123... --show-evidence --show-trace
# Markdown for PR comment
stella explain block sha256:abc123... --format markdown | gh pr comment 123 --body-file -
```
**Exit Codes:**
- `0` - Artifact is NOT blocked (all gates passed)
- `1` - Artifact IS blocked
- `2` - Error (not found, API error)
**Output (table):**
```
Artifact: sha256:abc123def456789012345678901234567890123456789012345678901234
Status: BLOCKED
Gate: VexTrust
Reason: Trust score below threshold (0.45 < 0.70)
Suggestion: Obtain VEX statement from trusted issuer
Evidence:
[VEX ] vex:sha256:de...23 vendor-x 2026-01-15T10:00:00Z
[REACH ] reach:sha256...56 static 2026-01-15T09:55:00Z
Replay: stella verify verdict --verdict urn:stella:verdict:sha256:abc123:v2.3.0:1737108000
```
**See Also:** [Explain Commands Documentation](explain.md)
---
## Additional Commands
### stella vuln query

333
docs/modules/telemetry/guides/p0-metrics.md Normal file
View File

@@ -0,0 +1,333 @@
# P0 Product Metrics
> **Sprint:** SPRINT_20260117_028_Telemetry_p0_metrics
> **Task:** P0M-007 - Documentation
This document describes the four P0 (highest priority) product-level metrics for tracking Stella Ops operational health.
## Overview
These metrics serve as the primary scoreboard for product health and should guide prioritization decisions. Per the AI Economics Moat advisory: "Prioritize work that improves them."
| Metric | Target | Alert Threshold |
|--------|--------|-----------------|
| Time to First Verified Release | P90 < 4 hours | P90 > 24 hours |
| Mean Time to Answer "Why Blocked" | P90 < 5 minutes | P90 > 1 hour |
| Support Minutes per Customer | Trend toward 0 | > 30 min/month |
| Determinism Regressions | Zero | Any policy-level |
---
## Metric 1: Time to First Verified Release
**Name:** `stella_time_to_first_verified_release_seconds`
**Type:** Histogram
### Definition
Elapsed time from fresh install (first service startup) to first successful verified promotion (policy gate passed, evidence recorded).
### Labels
| Label | Values | Description |
|-------|--------|-------------|
| `tenant` | (varies) | Tenant identifier |
| `deployment_type` | `fresh`, `upgrade` | Type of installation |
### Histogram Buckets
5m, 15m, 30m, 1h, 2h, 4h, 8h, 24h, 48h, 168h (1 week)
### Collection Points
1. **Install timestamp** - Recorded on first Authority service startup
2. **First promotion** - Recorded in Release Orchestrator on first verified promotion
### Why This Matters
A short time-to-first-release indicates:
- Good onboarding experience
- Clear documentation
- Sensible default configurations
- Working integrations
### Dashboard Usage
The Grafana dashboard shows:
- Histogram heatmap of time distribution
- P50/P90/P99 statistics
- Trend over time
### Alert Response
**Warning (P90 > 4 hours):**
1. Review recent onboarding experiences
2. Check for common configuration issues
3. Review documentation clarity
**Critical (P90 > 24 hours):**
1. Investigate blocked customers
2. Check for integration failures
3. Consider guided onboarding assistance
---
## Metric 2: Mean Time to Answer "Why Blocked"
**Name:** `stella_why_blocked_latency_seconds`
**Type:** Histogram
### Definition
Time from block decision to user viewing explanation (via CLI, UI, or API).
### Labels
| Label | Values | Description |
|-------|--------|-------------|
| `tenant` | (varies) | Tenant identifier |
| `surface` | `cli`, `ui`, `api` | Interface used to view explanation |
| `resolution_type` | `immediate`, `delayed` | Same session vs different session |
### Histogram Buckets
1s, 5s, 30s, 1m, 5m, 15m, 1h, 4h, 24h
### Collection Points
1. **Block decision** - Timestamp stored in verdict
2. **Explanation view** - Tracked when `stella explain block` or UI equivalent invoked
### Why This Matters
Short "why blocked" latency indicates:
- Clear block messaging
- Discoverable explanation tools
- Good explainability UX
Long latency may indicate:
- Users confused about where to find answers
- Documentation gaps
- UX friction
### Dashboard Usage
The Grafana dashboard shows:
- Histogram heatmap of latency distribution
- Trend line over time
- Breakdown by surface (CLI vs UI vs API)
### Alert Response
**Warning (P90 > 5 minutes):**
1. Review block notification messaging
2. Check CLI command discoverability
3. Verify UI links are prominent
**Critical (P90 > 1 hour):**
1. Investigate user flows
2. Add proactive notifications
3. Review documentation and help text
---
## Metric 3: Support Minutes per Customer
**Name:** `stella_support_burden_minutes_total`
**Type:** Counter
### Definition
Accumulated support time per customer per month. This is a manual/semi-automated metric for solo operations tracking.
### Labels
| Label | Values | Description |
|-------|--------|-------------|
| `tenant` | (varies) | Tenant identifier |
| `category` | `install`, `config`, `policy`, `integration`, `bug`, `other` | Support category |
| `month` | YYYY-MM | Month of support |
### Collection
Log support interactions using:
```bash
stella ops support log --tenant <id> --minutes <n> --category <cat>
```
Or via API:
```bash
POST /v1/ops/support/log
{
"tenant": "acme-corp",
"minutes": 15,
"category": "config"
}
```
### Why This Matters
This metric tracks operational scalability. For solo-scaled operations:
- Support burden should trend toward zero
- High support minutes indicate product gaps
- Categories identify areas needing improvement
### Dashboard Usage
The Grafana dashboard shows:
- Stacked bar chart by category
- Monthly trend per tenant
- Total support burden
### Alert Response
**Warning (> 30 min/month per tenant):**
1. Review support interactions for patterns
2. Identify documentation gaps
3. Create runbooks for common issues
**Critical (> 60 min/month per tenant):**
1. Escalate to product for feature work
2. Consider dedicated support time
3. Prioritize automation
---
## Metric 4: Determinism Regressions
**Name:** `stella_determinism_regressions_total`
**Type:** Counter
### Definition
Count of detected determinism failures in production (same inputs produced different outputs).
### Labels
| Label | Values | Description |
|-------|--------|-------------|
| `tenant` | (varies) | Tenant identifier |
| `component` | `scanner`, `policy`, `attestor`, `export` | Component with regression |
| `severity` | `bitwise`, `semantic`, `policy` | Fidelity tier of regression |
### Severity Tiers
| Tier | Description | Impact |
|------|-------------|--------|
| `bitwise` | Byte-for-byte output differs | Low - cosmetic |
| `semantic` | Output semantically differs | Medium - potential confusion |
| `policy` | Policy decision differs | **Critical** - audit risk |
### Collection Points
1. **Scheduled verification jobs** - Regular determinism checks
2. **Replay verification failures** - User-initiated replays
3. **CI golden test failures** - Development-time detection
### Why This Matters
Determinism is a core moat. Regressions indicate:
- Non-deterministic code introduced
- External dependency changes
- Time-sensitive logic bugs
**Policy-level regressions are audit-breaking** and must be fixed immediately.
### Dashboard Usage
The Grafana dashboard shows:
- Counter with severity breakdown
- Alert status indicator
- Historical trend
### Alert Response
**Warning (any bitwise/semantic):**
1. Review recent deployments
2. Check for dependency updates
3. Investigate affected component
**Critical (any policy):**
1. **Immediate investigation required**
2. Consider rollback
3. Review all recent policy decisions
4. Notify affected customers
---
## Dashboard Access
The P0 metrics dashboard is available at:
```
/grafana/d/stella-p0-metrics
```
Or directly:
```bash
stella ops dashboard p0
```
### Dashboard Features
- **Tenant selector** - Filter by specific tenant
- **Time range** - Adjust analysis window
- **SLO indicators** - Green/yellow/red status
- **Drill-down links** - Navigate to detailed views
---
## Alerting Configuration
Alerts are configured in `devops/telemetry/alerts/stella-p0-alerts.yml`.
### Alert Channels
Configure alert destinations in Grafana:
- Slack/Teams for warnings
- PagerDuty for critical alerts
- Email for summaries
### Silencing Alerts
During maintenance windows:
```bash
stella ops alerts silence --duration 2h --reason "Planned maintenance"
```
---
## Implementation Notes
### Source Files
| Component | Location |
|-----------|----------|
| Metric definitions | `src/Telemetry/StellaOps.Telemetry.Core/P0ProductMetrics.cs` |
| Install timestamp | `src/Telemetry/StellaOps.Telemetry.Core/InstallTimestampService.cs` |
| Dashboard template | `devops/telemetry/grafana/dashboards/stella-ops-p0-metrics.json` |
| Alert rules | `devops/telemetry/alerts/stella-p0-alerts.yml` |
### Adding Custom Metrics
To add additional P0-level metrics:
1. Define in `P0ProductMetrics.cs`
2. Add collection points in relevant services
3. Create dashboard panel in Grafana JSON
4. Add alert rules
5. Update this documentation
---
## Related
- [Observability Guide](observability.md)
- [Alerting Configuration](alerting.md)
- [Runbook: Metric Collection Issues](../../operations/runbooks/telemetry-metrics-ops.md)
---
_Last updated: 2026-01-17 (UTC)_