First-time user experience fixes and platform contract repairs
FTUX fixes (Sprint 316-001): - Remove all hardcoded fake data from dashboard — fresh installs show honest setup guide instead of fake crisis data (5 fake criticals gone) - Curate advisory source defaults: 32 sources disabled by default (ecosystem, geo-restricted, exploit, hardware, mirror). ~43 core sources remain enabled. StellaOps Mirror no longer enabled at priority 1. - Filter Mirror-category sources from Create Domain wizard to prevent circular mirror-from-mirror chains - Add 404 catch-all route — unknown URLs show "Page Not Found" instead of silently rendering the dashboard - Fix arrow characters in release target path dropdown (? → →) - Add login credentials to quickstart documentation - Update Feature Matrix: 14 release orchestration features marked as shipped (was marked planned) Platform contract repairs (from prior session): - Add /api/v1/jobengine/quotas/summary endpoint on Platform - Fix gateway route prefix matching for /policy/shadow/* and /policy/simulations/* (regex routes instead of exact match) - Fix VexHub PostgresVexSourceRepository missing interface method - Fix advisory-vex-sources sweep text expectation - Fix mirror operator journey auth (session storage token extraction) Verified: 110/111 canonical routes passing (1 unrelated stale approval ref) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
master
185
docs/qa/DEEP_ADVISORY_MIRROR_AUDIT_20260316.md
Normal file
185
docs/qa/DEEP_ADVISORY_MIRROR_AUDIT_20260316.md
Normal file
@@ -0,0 +1,185 @@
|
||||
# Deep Advisory & Mirror Audit — Stella Ops
|
||||
|
||||
**Date**: 2026-03-16
|
||||
**Method**: Interactive Playwright walkthrough + source code analysis
|
||||
**Focus**: Advisory source catalog, mirror architecture, mode confusion, first-time operator experience
|
||||
|
||||
---
|
||||
|
||||
## The Core Problem: Mirror Source vs Mirror Mode Are Disconnected
|
||||
|
||||
Stella Ops has three advisory operating modes:
|
||||
- **Direct**: Fetch advisories from upstream sources (NVD, OSV, GHSA, etc.) directly
|
||||
- **Mirror**: Consume pre-aggregated advisory bundles from an upstream Stella Ops mirror
|
||||
- **Hybrid**: Both direct and mirror sources active
|
||||
|
||||
The mirror feature has two roles:
|
||||
- **Producer**: Create mirror domains that bundle advisories for downstream consumers
|
||||
- **Consumer**: Connect to an upstream mirror and pull bundles
|
||||
|
||||
**The problem**: The advisory source catalog and the mirror mode are independent systems that don't coordinate:
|
||||
|
||||
### Issue 1: StellaOps Mirror Source Enabled by Default in Direct Mode
|
||||
|
||||
**File**: `src/Concelier/__Libraries/StellaOps.Concelier.Core/Sources/SourceDefinitions.cs:1351-1366`
|
||||
|
||||
The `StellaMirror` source definition has:
|
||||
```csharp
|
||||
BaseEndpoint = "https://mirror.stella-ops.org/api/v1",
|
||||
DefaultPriority = 1, // HIGHEST priority
|
||||
EnabledByDefault = true // inherited from base class
|
||||
```
|
||||
|
||||
On a fresh self-hosted install:
|
||||
- Mode badge shows **"Direct"**
|
||||
- But "StellaOps Mirror" source is **enabled** at **priority 1** (highest)
|
||||
- Its endpoint `mirror.stella-ops.org` is an external SaaS URL that doesn't exist for self-hosted
|
||||
- Health check fails: "SSL/TLS error connecting to StellaOps Mirror"
|
||||
|
||||
**Impact**: The highest-priority advisory source is a broken external URL. If it were reachable, it would override all local direct sources. The system silently operates with a failed priority-1 source.
|
||||
|
||||
**Fix**: Either:
|
||||
- Set `EnabledByDefault = false` for the StellaMirror source
|
||||
- Or: gate it behind mirror mode — only enable when mode is "Mirror" or "Hybrid"
|
||||
- Or: replace the hardcoded URL with a placeholder that requires explicit configuration
|
||||
|
||||
### Issue 2: Mirror Source Appears in Create Mirror Domain Wizard
|
||||
|
||||
When creating a new mirror domain (producer role), the source selection wizard lists ALL sources including "StellaOps Mirror" under the Mirror (1) category. An operator can select it and create a mirror domain that sources from... the broken external mirror.
|
||||
|
||||
**Impact**: Creates a circular or broken dependency chain. A mirror domain that sources from a nonexistent upstream.
|
||||
|
||||
**Fix**: Exclude sources of type `StellaMirror` from the Create Mirror Domain source picker. A mirror domain should only aggregate direct-fetch sources, not other mirrors.
|
||||
|
||||
### Issue 3: No Relationship Between Catalog Toggle and Mirror Consumer Setup
|
||||
|
||||
Two separate ways to "use a mirror" exist:
|
||||
1. **Catalog toggle**: Enable "StellaOps Mirror" source in the advisory catalog → hardcoded to `mirror.stella-ops.org`
|
||||
2. **Mirror Consumer Wizard**: "Connect to Mirror" → 4-step wizard where you enter a custom mirror URL
|
||||
|
||||
These are independent. Enabling the catalog mirror source does NOT invoke the consumer wizard. Completing the consumer wizard does NOT update the catalog mirror source's endpoint.
|
||||
|
||||
**Impact**: A user who completes the consumer wizard and enters `https://my-corp-mirror.internal` still has the catalog's mirror source pointing at `mirror.stella-ops.org`. Conversely, a user who toggles the catalog source thinks they've configured mirror mode, but the system mode stays "Direct".
|
||||
|
||||
**Fix**: Unify these:
|
||||
- When the mirror consumer wizard is completed, it should: (a) update the StellaMirror source endpoint to the configured URL, (b) enable the source, (c) switch mode to "Mirror" or "Hybrid"
|
||||
- The catalog's StellaMirror source toggle should redirect to the consumer wizard if no mirror is configured
|
||||
- Or: remove the StellaMirror source from the catalog entirely and make it purely managed through the mirror dashboard
|
||||
|
||||
### Issue 4: "Direct" Mode Label Is Misleading When Mirror Source Is Enabled
|
||||
|
||||
The mirror context header shows "Direct" (green badge) but a mirror source is enabled at highest priority. The mode badge doesn't reflect actual source configuration.
|
||||
|
||||
**Fix**: Derive the mode from actual source state:
|
||||
- If only direct sources are enabled → "Direct"
|
||||
- If only mirror source is enabled → "Mirror"
|
||||
- If both → "Hybrid"
|
||||
Or at minimum, show a warning: "Mirror source is enabled but unreachable"
|
||||
|
||||
---
|
||||
|
||||
## Advisory Source Health: Fresh Install Reality
|
||||
|
||||
After "Check All" on a fresh install:
|
||||
- **55 healthy** (sources reachable from Docker network)
|
||||
- **18 failed** (unreachable from Docker network or nonexistent)
|
||||
|
||||
**Failed sources by category:**
|
||||
| Source | Likely Reason |
|
||||
|--------|--------------|
|
||||
| Oracle Security | DNS/network from Docker |
|
||||
| npm/PyPI/RubyGems/Maven/Packagist/Hex.pm Advisories (6) | OSV-based, likely endpoint differences |
|
||||
| CERT-In (India) | Geo-restricted or slow |
|
||||
| FSTEC BDU (Russia) | Geo-restricted |
|
||||
| CSAF Aggregator | Endpoint format |
|
||||
| VEX Hub | Not a real endpoint yet |
|
||||
| MITRE D3FEND | Endpoint format |
|
||||
| Exploit-DB | Likely rate-limited |
|
||||
| Docker Official CVEs | Endpoint format |
|
||||
| AMD Security | DNS/network |
|
||||
| Siemens ProductCERT | Endpoint format |
|
||||
| Ruby Advisory DB | Endpoint format |
|
||||
| StellaOps Mirror | Nonexistent external URL |
|
||||
|
||||
**Finding**: 74 out of 75 sources are enabled by default (`EnabledByDefault = true`). On a fresh install, 18 immediately fail health checks. The stats bar shows "55 enabled, 55 healthy, 19 failed" — but this is confusing because "55 enabled" should be 74 (or 75). The enabled count in the stats bar and the actual enabled count don't match.
|
||||
|
||||
**Fix**:
|
||||
- Don't enable all 75 sources by default. Enable a curated set (Primary 4 + major vendors + major distributions = ~20-25 sources)
|
||||
- Disable ecosystem-specific sources (npm, PyPI, etc.) by default — let users enable for their stack
|
||||
- Disable geo-restricted sources (FSTEC, NKCKI) by default
|
||||
- Show a "Recommended for your platform" selection during first-time setup
|
||||
|
||||
---
|
||||
|
||||
## Mirror Domain Builder: Good UX, Missing Guardrails
|
||||
|
||||
The 3-step domain builder wizard is well-designed:
|
||||
1. **Select Sources** — categorized picker with quick-select buttons, search, selection summary
|
||||
2. **Configure Domain** — auto-generated ID/name, 5 export formats (JSON/JSONL/OpenVEX/CSAF/CycloneDX), rate limits, auth, signing
|
||||
3. **Review & Create** — summary + JSON filter preview + "generate immediately" option
|
||||
|
||||
**Good**:
|
||||
- Auto-generated domain ID from selection (`mirror-primary-4src`)
|
||||
- Multiple export formats including OpenVEX and CSAF
|
||||
- Rate limiting per domain
|
||||
- Optional signing and authentication
|
||||
- "Generate immediately" checkbox
|
||||
|
||||
**Issues**:
|
||||
- Can select "StellaOps Mirror" as a source (circular, see Issue 2)
|
||||
- No indication of which sources are actually healthy — I might build a domain from 4 sources where 2 are failing
|
||||
- No estimate of bundle size or generation time
|
||||
|
||||
---
|
||||
|
||||
## Mirror Consumer Wizard: Clean but Disconnected
|
||||
|
||||
The 4-step consumer wizard:
|
||||
1. **Connect** — Enter mirror base URL + test connection
|
||||
2. **Signature** — Verify signing key
|
||||
3. **Sync & Mode** — Schedule + mode selection
|
||||
4. **Review & Activate** — Confirm
|
||||
|
||||
**Good**:
|
||||
- Proper signature verification step
|
||||
- "Test Connection" button
|
||||
- Mode selection in the workflow
|
||||
|
||||
**Issues**:
|
||||
- Completing this wizard doesn't update the catalog's StellaMirror source (see Issue 3)
|
||||
- Placeholder URL `https://mirror.stella-ops.org` is the same broken SaaS URL as the catalog source
|
||||
- No way to test with the local instance's own mirror domains (self-mirroring for verification)
|
||||
|
||||
---
|
||||
|
||||
## Recommended Architecture Changes
|
||||
|
||||
### Short-term (S/M effort):
|
||||
1. **Set `EnabledByDefault = false` for StellaMirror source** — prevents broken priority-1 source on fresh install
|
||||
2. **Exclude StellaMirror from Create Domain wizard** — prevents circular mirror chains
|
||||
3. **Curate default-enabled sources** — only enable Primary + top 15 vendor/distribution sources by default
|
||||
4. **Fix stats bar count** — "enabled" count should match actual enabled sources
|
||||
5. **Show health status in Create Domain source picker** — badge healthy/failed next to each source
|
||||
|
||||
### Medium-term (L effort):
|
||||
6. **Unify catalog mirror source and consumer wizard** — catalog toggle should invoke wizard, wizard should update catalog
|
||||
7. **Derive mode from source state** — "Direct"/"Mirror"/"Hybrid" computed from actual enabled sources
|
||||
8. **Add "Recommended sources" first-run flow** — during setup, suggest sources based on detected platform (Docker → Container sources, Python → PyPI, etc.)
|
||||
|
||||
### Long-term (XL effort):
|
||||
9. **Mirror domain ↔ consumer federation** — Instance A's domain auto-discoverable by Instance B's consumer wizard via DNS SRV or well-known URL
|
||||
10. **Self-test mirror consumer** — "Connect to localhost" option that validates the producer flow by consuming from own mirror domains
|
||||
|
||||
---
|
||||
|
||||
## Files to Modify
|
||||
|
||||
| Fix | File | Change |
|
||||
|-----|------|--------|
|
||||
| #1 | `src/Concelier/__Libraries/StellaOps.Concelier.Core/Sources/SourceDefinitions.cs:1351` | Add `EnabledByDefault = false` to StellaMirror |
|
||||
| #2 | `src/Web/StellaOps.Web/src/app/features/integrations/advisory-vex-sources/mirror-domain-builder.component.ts` | Filter out sources where `category === 'Mirror'` from the picker |
|
||||
| #3 | `src/Concelier/__Libraries/StellaOps.Concelier.Core/Sources/SourceDefinitions.cs` | Set `EnabledByDefault = false` on ecosystem/geo-restricted sources |
|
||||
| #4 | `src/Web/StellaOps.Web/src/app/features/integrations/advisory-vex-sources/advisory-source-catalog.component.ts` | Fix enabled count computation |
|
||||
| #5 | `src/Web/StellaOps.Web/src/app/features/integrations/advisory-vex-sources/mirror-domain-builder.component.ts` | Show health badge next to source names |
|
||||
| #6 | Multiple files | Wire consumer wizard completion to update catalog source + mode |
|
||||
| #7 | `src/Concelier/StellaOps.Concelier.WebService/Extensions/MirrorDomainManagementEndpointExtensions.cs` | Compute mode from source state |
|
||||
Reference in New Issue
Block a user