9.6 KiB
9.6 KiB
CLI Batch A -- E2E Test Results
Date: 2026-02-15
Agent: batch-a
CLI Project: src/Cli/StellaOps.Cli/StellaOps.Cli.csproj
Configuration: Release (pre-built, --no-build)
Environment note: SM remote probe fails (expected -- no SM remote service running). Adds ~4s startup latency per invocation.
Top-Level Command Summary
| # | Command | Description | Subcommands | --help OK | Behavioral Test | Exit Code | Notes |
|---|---|---|---|---|---|---|---|
| 1 | scanner |
Manage scanner artifacts and lifecycle | download, workers |
YES | N/A (container-dependent) | 0 | 2 subcommands |
| 2 | scan |
Execute scanners and manage scan outputs | entrytrace, sarif, replay, gate-policy, gate-results, layers, layer-sbom, recipe, diff, delta, verify-patches, download, workers, secrets, image, run, upload, graph |
YES | N/A (requires scan data) | 0 | 18 subcommands -- richest command |
| 3 | image |
OCI image operations | inspect |
YES | N/A (requires registry) | 0 | 1 subcommand |
| 4 | ruby |
Work with Ruby analyzer outputs | inspect, resolve |
YES | ruby inspect --help OK |
0 | 2 subcommands |
| 5 | php |
Work with PHP analyzer outputs | inspect |
YES | N/A | 0 | 1 subcommand |
| 6 | python |
Work with Python analyzer outputs | inspect |
YES | N/A | 0 | 1 subcommand |
| 7 | bun |
Work with Bun analyzer outputs | inspect, resolve |
YES | N/A | 0 | 2 subcommands |
| 8 | db |
Trigger Concelier database operations | fetch, merge, export |
YES | N/A (requires backend) | 0 | 3 subcommands |
| 9 | sources |
Interact with source ingestion workflows | ingest, list, check, enable, disable, status |
YES | sources list CRASH (exit 1), sources status CRASH (exit 1) |
0 (help) / 1 (run) | BUG: ISourceRegistry not registered in DI |
| 10 | aoc |
Aggregation-Only Contract verification | verify |
YES | aoc verify exits 71 (tenant required) |
0 (help) / 71 (run) | Correct error: requires --tenant |
| 11 | auth |
Manage authentication | login, logout, status, whoami, revoke, token |
YES | auth status exits 1 (authority not configured) |
0 (help) / 1 (run) | Expected: no Authority URL configured |
| 12 | tenants |
Manage tenant contexts | list, use, current, clear |
YES | tenants current exits 0: "No active tenant configured." |
0 | Correct offline behavior |
| 13 | policy |
Interact with Policy Engine | simulate, activate, lint, edit, test, new, history, explain, init, compile, version, submit, review, publish, rollback, sign, verify-signature, lattice, verdicts, promote, validate-yaml, install, list-packs, export, import, validate, evaluate |
YES | policy lint /nonexistent.stella exits 4 (file not found) |
0 (help) / 4 (lint) | 27 subcommands; correct error for missing file |
| 14 | tools |
Local policy tooling | policy-dsl-validate, policy-schema-export, policy-simulation-smoke, lint, benchmark, migrate |
YES | N/A | 0 | 6 subcommands; benchmark has sub-subs (policy/scan/crypto) |
| 15 | task-runner |
Interact with Task Runner | simulate |
YES | N/A | 0 | 1 subcommand |
| 16 | findings |
Inspect policy findings | ls, get, explain |
YES | findings ls exits 1 (--policy required) |
0 (help) / 1 (run) | Correct: shows required option hint |
| 17 | advise |
Advisory AI pipelines | run, summarize, explain, remediate, batch, open-pr, ask, chat-doctor, chat-settings, export |
YES | advise run --help OK |
0 | 10 subcommands |
| 18 | config |
Manage configuration | show, list, notify, integrations, feeds, registry, sources, signals |
YES | config show exits 0 (shows defaults), config list exits 0 (lists paths) |
0 | 8 subcommands; behavioral tests pass |
| 19 | kms |
Manage signing keys | export, import |
YES | Both --help OK |
0 | 2 subcommands |
| 20 | key |
Key management | list, add, revoke, rotate, status, history, verify |
YES | N/A (requires anchorId) | 0 | 7 subcommands |
| 21 | issuer |
Issuer key management | keys (sub: list, create, rotate, revoke) |
YES | issuer keys --help OK |
0 | Nested: keys has 4 sub-subcommands |
Subcommand --help Verification
| Parent | Subcommand | --help OK | Exit Code | Notes |
|---|---|---|---|---|
scanner |
download |
YES | 0 | Options: --channel, --output, --overwrite, --no-install |
scanner |
workers |
YES | 0 | Sub-subcommands: get, set |
scan |
entrytrace |
YES | 0 | Options: --scan-id (required), --include-ndjson, --semantic |
scan |
sarif |
YES | 0 | Options: --scan-id (required), -o, --pretty, --include-hardening, --include-reachability, --min-severity |
scan |
replay |
YES | 0 | Options: --artifact (req), --manifest (req), --feeds (req), --policy (req), --offline, --verify-inputs |
scan |
secrets |
YES | 0 | Sub-subcommand: bundle |
scan |
graph |
YES | 0 | Options: --lang (req), --target (req), --format, --upload, --include-tests |
image |
inspect |
YES | 0 | Options: -r, -l, -p platform, -o format, --timeout |
auth |
login |
YES | 0 | Options: --force |
auth |
status |
YES | 0 | No extra options |
auth |
whoami |
YES | 0 | No extra options |
db |
fetch |
YES | 0 | Options: --source (req), --stage, --mode |
db |
merge |
YES | 0 | No extra options |
db |
export |
YES | 0 | Options: --format, --delta, --publish-full, --publish-delta, --bundle-full, --bundle-delta |
policy |
lint |
YES | 0 | Args: file; Options: -f, -o |
policy |
new |
YES | 0 | Args: name; Options: -t template, -o, -d, --tag, --shadow, --fixtures, --git-init |
policy |
compile |
YES | 0 | Args: file; Options: -o, --no-ir, --no-digest, --optimize, --strict |
policy |
validate-yaml |
YES | 0 | Args: path; Options: --schema, --strict |
policy |
list-packs |
YES | 0 | Options: --source |
policy |
evaluate |
YES | 0 | Options: -p policy (req), -i input (req), --format, -e environment, --include-remediation |
tenants |
list |
YES | 0 | Options: --tenant, --json |
tenants |
use |
YES | 0 | Args: tenant-id |
tenants |
clear |
YES | 0 | No extra options |
tools |
lint |
YES | 0 | Options: -i input (req), --fix, --strict, -f format |
tools |
benchmark |
YES | 0 | Sub-subcommands: policy, scan, crypto |
tools |
migrate |
YES | 0 | Sub-subcommands: config, data |
task-runner |
simulate |
YES | 0 | Options: --manifest, --inputs, --format, --output |
kms |
export |
YES | 0 | Options: --root, --key-id (req), --version, --output (req), --force, --passphrase |
kms |
import |
YES | 0 | Options: --root, --key-id (req), --input (req), --version, --passphrase |
issuer |
keys |
YES | 0 | Sub-subcommands: list, create, rotate, revoke |
advise |
run |
YES | 0 | Args: task; Options: --advisory-key (req), many more |
findings |
ls |
YES (via error) | 1 | Shows help with required --policy hint |
config |
show |
YES | 0 | No extra options |
Behavioral Test Results
| Command | Invocation | Exit Code | Behavior | Verdict |
|---|---|---|---|---|
auth status |
auth status |
1 | "Authority URL not configured. Set STELLAOPS_AUTHORITY_URL and run 'auth login'." | PASS -- correct error |
tenants current |
tenants current |
0 | "No active tenant configured. Use 'stella tenants use ' to set one." | PASS -- correct offline |
config show |
config show |
0 | Shows all config keys with defaults (Backend URL, Concelier URL, API Key, etc.) | PASS -- works offline |
config list |
config list |
0 | Lists all config paths grouped by section (notify, feeds, integrations, etc.) | PASS -- works offline |
sources list |
sources list |
1 | CRASH: InvalidOperationException: No service for type 'ISourceRegistry' has been registered. |
FAIL -- DI bug |
sources status |
sources status |
1 | CRASH: Same ISourceRegistry DI exception |
FAIL -- DI bug |
aoc verify |
aoc verify |
71 | "Tenant must be provided via --tenant or STELLA_TENANT." | PASS -- correct validation |
policy lint |
policy lint /nonexistent.stella |
4 | "Error: Policy file not found: .../nonexistent.stella" | PASS -- correct file-not-found |
findings ls |
findings ls |
1 | "Option '--policy' is required." + help text | PASS -- correct validation |
Bugs Found
BUG-001: sources list and sources status crash with DI exception
Severity: Medium
Commands affected: sources list, sources status
Error: System.InvalidOperationException: No service for type 'StellaOps.Concelier.Core.Sources.ISourceRegistry' has been registered.
Location: src/Cli/StellaOps.Cli/Commands/Sources/SourcesCommandHandlers.cs:line 35 (list), line 332 (status)
Root cause: The ISourceRegistry service is not registered in the CLI's DI container. The sources --help works fine, but actual invocation fails.
Impact: Users cannot list or check status of advisory sources via CLI without backend connectivity.
Summary
- 21/21 commands have working
--help(exit 0) - All subcommand --help tests pass (30+ subcommands tested)
- 9 behavioral tests run: 7 PASS, 2 FAIL
- 1 bug found:
sources list/sources statusDI registration missing forISourceRegistry - Total subcommands discovered: 100+ across all 21 top-level commands
- Richest commands:
policy(27 subcmds),scan(18 subcmds),advise(10 subcmds),config(8 subcmds)