git.stella-ops.org/docs/operations/trust-lattice-troubleshooting.md

# Trust Lattice Troubleshooting Guide

> **Version**: 1.0.0
> **Last Updated**: 2025-12-22
> **Audience**: Support and Development teams

---

## Quick Reference

| Symptom | Likely Cause | Section |
|---------|--------------|---------|
| Low confidence scores | Stale VEX data or missing sources | [2.1](#21-low-confidence-scores) |
| Gate failures blocking builds | Threshold too high or source issues | [2.2](#22-gate-failures) |
| Verdict replay mismatches | Non-deterministic inputs | [2.3](#23-verdict-replay-failures) |
| Unexpected trust changes | Calibration drift | [2.4](#24-calibration-issues) |
| Conflicting verdicts | Multi-source disagreement | [2.5](#25-claim-conflicts) |

---

## 1. Diagnostic Commands

### 1.1 Check System Health

```bash
# Excititor health
curl https://api.example.com/excititor/health

# Policy Engine health
curl https://api.example.com/policy/health

# Authority health
curl https://api.example.com/authority/health
```

### 1.2 Trace a Verdict

```bash
# Get detailed verdict explanation
stella verdict explain <manifestId>

# Output includes:
# - All claims considered
# - Trust vector scores
# - Strength/freshness multipliers
# - Gate evaluation results
# - Conflict detection
```

### 1.3 Check VEX Source Status

```bash
# List all sources with status
stella vex source list

# Check specific source
stella vex source status vendor:redhat

# Sample output:
# Source: vendor:redhat
# Status: healthy
# Last fetch: 2025-12-22T10:00:00Z
# Documents: 15234
# Freshness: 2.3 hours
```

---

## 2. Common Issues

### 2.1 Low Confidence Scores

**Symptoms:**
- Verdicts have confidence < 0.5
- Many "under_investigation" statuses

**Diagnosis:**

1. Check claim freshness:
   ```bash
   stella claim analyze --cve CVE-2025-12345 --asset sha256:abc123

   # Look for:
   # - Freshness multiplier < 0.5 (claim older than 180 days)
   # - No high-trust sources
   ```

2. Check trust vector values:
   ```bash
   stella trustvector show vendor:redhat

   # Low scores indicate:
   # - Signature verification issues (P)
   # - Poor scope matching (C)
   # - Non-deterministic outputs (R)
   ```

3. Check for missing VEX coverage:
   ```bash
   stella vex coverage --purl pkg:npm/lodash@4.17.21

   # No claims? Source may not cover this package
   ```

**Resolution:**

- If freshness is low: Check if source is publishing updates
- If trust vector is low: Review source verification settings
- If coverage is missing: Add additional VEX sources

### 2.2 Gate Failures

**Symptoms:**
- Builds failing with "Gate: MinimumConfidenceGate FAILED"
- Policy violations despite VEX claims

**Diagnosis:**

1. Check gate thresholds:
   ```bash
   stella gates show minimumConfidence

   # Thresholds:
   #   production: 0.75
   #   staging: 0.60
   #   development: 0.40
   ```

2. Compare with verdict confidence:
   ```bash
   stella verdict show <manifestId> | grep confidence

   # confidence: 0.68  <- Below 0.75 production threshold
   ```

3. Check which gate failed:
   ```bash
   stella verdict gates <manifestId>

   # Gates:
   #   MinimumConfidenceGate: FAILED (0.68 < 0.75)
   #   SourceQuotaGate: PASSED
   #   UnknownsBudgetGate: PASSED
   ```

**Resolution:**

- Temporary: Lower threshold (with approval)
- Long-term: Add corroborating VEX sources
- If single-source: Check SourceQuotaGate corroboration

### 2.3 Verdict Replay Failures

**Symptoms:**
- Replay verification returns success: false
- Audit failures due to non-determinism

**Diagnosis:**

1. Get detailed diff:
   ```bash
   stella verdict replay --diff <manifestId>

   # Differences:
   #   result.confidence: 0.82 -> 0.79
   #   inputs.vexDocumentDigests[2]: sha256:abc... (missing)
   ```

2. Common causes:

   | Difference | Likely Cause |
   |------------|--------------|
   | VEX digest mismatch | Document was modified after verdict |
   | Confidence delta | Clock cutoff drift (freshness calc) |
   | Missing claims | Source was unavailable during replay |
   | Different status | Policy version changed |

3. Check input availability:
   ```bash
   # Verify all pinned inputs exist
   stella cas verify --digest sha256:abc123
   ```

**Resolution:**

- Clock drift: Ensure NTP synchronization across nodes
- Missing inputs: Restore from backup or acknowledge drift
- Policy change: Compare policy hashes between original and replay

### 2.4 Calibration Issues

**Symptoms:**
- Trust vectors changed unexpectedly
- Accuracy metrics declining

**Diagnosis:**

1. Review recent calibrations:
   ```bash
   stella calibration history vendor:redhat --epochs 5

   # Epoch 42: accuracy=0.92, delta=(-0.02, +0.02, 0)
   # Epoch 41: accuracy=0.94, delta=(-0.01, +0.01, 0)
   ```

2. Check comparison results:
   ```bash
   stella calibration epoch 42 --details

   # Total claims: 1500
   # Correct: 1380
   # False positives: 45
   # False negatives: 75
   # Detected bias: OptimisticBias
   ```

3. Check for data quality issues:
   ```bash
   # Look for corrupted truth data
   stella calibration validate-truth --epoch 42
   ```

**Resolution:**

- High false positive: Reduce provenance score
- High false negative: Review coverage matching
- Data quality issue: Re-run with corrected truth set
- Emergency: Rollback to previous epoch

### 2.5 Claim Conflicts

**Symptoms:**
- Verdicts show hasConflicts: true
- Confidence reduced due to conflict penalty

**Diagnosis:**

1. View conflict details:
   ```bash
   stella verdict conflicts <manifestId>

   # Conflicts:
   #   vendor:redhat claims: not_affected
   #   hub:osv claims: affected
   #   Conflict penalty applied: 0.25
   ```

2. Investigate source disagreement:
   ```bash
   # Get raw claims from each source
   stella vex claim --source vendor:redhat --cve CVE-2025-12345
   stella vex claim --source hub:osv --cve CVE-2025-12345
   ```

3. Check claim timestamps:
   ```bash
   # Older claim may be outdated
   stella claim compare vendor:redhat hub:osv --cve CVE-2025-12345
   ```

**Resolution:**

- If one source is stale: Flag for review
- If genuine disagreement: Higher-trust source wins (by design)
- If persistent: Consider source override in policy

---

## 3. Performance Issues

### 3.1 Slow Claim Scoring

**Symptoms:**
- Scoring latency > 100ms
- Timeouts during high load

**Diagnosis:**

```bash
# Check scoring performance
stella perf scoring --samples 100

# Look for:
# - Cache miss rate
# - Trust vector lookups
# - Freshness calculation overhead
```

**Resolution:**

- Enable trust vector caching
- Pre-compute freshness for common cutoffs
- Scale Excititor horizontally

### 3.2 Slow Verdict Replay

**Symptoms:**
- Replay verification > 5 seconds
- Timeout during audit

**Diagnosis:**

```bash
# Check input retrieval time
stella verdict replay --timing <manifestId>

# Timing:
#   Input fetch: 3.2s
#   Score compute: 0.1s
#   Merge: 0.05s
#   Total: 3.35s
```

**Resolution:**

- Ensure CAS storage is local or cached
- Pre-warm verdict cache for critical assets
- Increase timeout for large manifests

---

## 4. Integration Issues

### 4.1 VEX Source Not Recognized

**Symptoms:**
- Claims from source not included in verdicts
- Source shows as "unknown" class

**Resolution:**

1. Register source in configuration:
   ```yaml
   # etc/trust-lattice.yaml
   sources:
     - id: vendor:newvendor
       class: vendor
       trustVector:
         provenance: 0.85
         coverage: 0.70
         replayability: 0.60
   ```

2. Reload configuration:
   ```bash
   stella config reload --service excititor
   ```

### 4.2 Gate Not Evaluating

**Symptoms:**
- Expected gate not appearing in results
- Gate shows as "disabled"

**Resolution:**

1. Check gate configuration:
   ```bash
   stella gates list --show-disabled
   ```

2. Enable gate:
   ```yaml
   # etc/policy-gates.yaml
   gates:
     minimumConfidence:
       enabled: true  # Ensure this is true
   ```

---

## 5. Support Information

### 5.1 Collecting Diagnostic Bundle

```bash
stella support bundle --include trust-lattice \
  --since 1h --output /tmp/diag.zip
```

Bundle includes:
- Trust vector snapshots
- Recent verdicts
- Gate evaluations
- Calibration history
- System metrics

### 5.2 Log Locations

| Service | Log Path |
|---------|----------|
| Excititor | `/var/log/stellaops/excititor.log` |
| Policy | `/var/log/stellaops/policy.log` |
| Authority | `/var/log/stellaops/authority.log` |

### 5.3 Contact

- **Support**: support@stella-ops.org
- **Documentation**: `docs/modules/excititor/trust-lattice.md`
- **GitHub Issues**: https://github.com/stella-ops/stella-ops/issues

---

*Document Version: 1.0.0*
*Sprint: 7100.0003.0002*