git.stella-ops.org/docs/guides/keyless-signing-troubleshooting.md

# Keyless Signing Troubleshooting Guide

This guide covers common issues when integrating StellaOps keyless signing into CI/CD pipelines.

## Common Errors

### OIDC Token Acquisition Failures

#### Error: "Unable to get OIDC token"

**Symptoms:**
```
Error: Unable to get ACTIONS_ID_TOKEN_REQUEST_URL
```

**Cause:** The workflow doesn't have `id-token: write` permission.

**Solution:**
```yaml
# GitHub Actions
permissions:
  id-token: write
  contents: read

# GitLab CI
job:
  id_tokens:
    STELLAOPS_OIDC_TOKEN:
      aud: sigstore
```

#### Error: "Token audience mismatch"

**Symptoms:**
```
Error: Token audience 'api://default' does not match expected 'sigstore'
```

**Cause:** OIDC token was requested with wrong audience.

**Solution:**
```yaml
# GitHub Actions
OIDC_TOKEN=$(curl -sLS "${ACTIONS_ID_TOKEN_REQUEST_URL}&audience=sigstore" \
  -H "Authorization: bearer ${ACTIONS_ID_TOKEN_REQUEST_TOKEN}")

# GitLab CI
id_tokens:
  STELLAOPS_OIDC_TOKEN:
    aud: sigstore  # Must be 'sigstore' for Fulcio
```

---

### Fulcio Certificate Errors

#### Error: "Failed to get certificate from Fulcio"

**Symptoms:**
```
Error: error getting certificate from Fulcio: 401 Unauthorized
```

**Causes:**
1. OIDC token expired (tokens are short-lived, typically 5-10 minutes)
2. Fulcio doesn't recognize the OIDC issuer
3. Network connectivity issues to Fulcio

**Solutions:**

1. **Token expiry:** Request token immediately before signing:
```yaml
- name: Get OIDC Token
  id: oidc
  run: |
    # Get fresh token right before signing
    OIDC_TOKEN=$(curl -sLS "${ACTIONS_ID_TOKEN_REQUEST_URL}&audience=sigstore" \
      -H "Authorization: bearer ${ACTIONS_ID_TOKEN_REQUEST_TOKEN}" \
      | jq -r '.value')
    echo "token=${OIDC_TOKEN}" >> $GITHUB_OUTPUT

- name: Sign (immediately after)
  env:
    STELLAOPS_OIDC_TOKEN: ${{ steps.oidc.outputs.token }}
  run: stella attest sign --keyless --artifact "$DIGEST"
```

2. **Unknown issuer:** Ensure your CI platform is supported:
   - GitHub Actions: `https://token.actions.githubusercontent.com`
   - GitLab.com: `https://gitlab.com`
   - Self-hosted GitLab: Must be configured in Fulcio

3. **Network issues:** Check connectivity:
```bash
curl -v https://fulcio.sigstore.dev/api/v2/signingCert
```

#### Error: "Certificate identity not found in token"

**Symptoms:**
```
Error: no matching subject or SAN found in OIDC token
```

**Cause:** Token claims don't include expected identity fields.

**Solution:** Verify token contents:
```bash
# Decode and inspect token (don't do this in production logs)
echo $OIDC_TOKEN | cut -d. -f2 | base64 -d | jq .
```

Expected claims for GitHub Actions:
```json
{
  "sub": "repo:org/repo:ref:refs/heads/main",
  "iss": "https://token.actions.githubusercontent.com",
  "repository": "org/repo",
  "ref": "refs/heads/main"
}
```

---

### Rekor Transparency Log Errors

#### Error: "Failed to upload to Rekor"

**Symptoms:**
```
Error: error uploading entry to Rekor: 500 Internal Server Error
```

**Causes:**
1. Rekor service temporarily unavailable
2. Entry too large
3. Network issues

**Solutions:**

1. **Retry with backoff:**
```yaml
- name: Sign with retry
  run: |
    for i in 1 2 3; do
      stella attest sign --keyless --artifact "$DIGEST" && break
      echo "Attempt $i failed, retrying in 30s..."
      sleep 30
    done
```

2. **Check Rekor status:** https://status.sigstore.dev/

3. **Use offline bundle (air-gapped):**
```bash
stella attest sign --keyless --artifact "$DIGEST" --offline-bundle
```

#### Error: "Rekor entry not found"

**Symptoms:**
```
Error: entry not found in transparency log
```

**Cause:** Verification requiring Rekor but entry wasn't logged (offline signing).

**Solution:** Either:
- Sign with Rekor enabled (default)
- Verify without Rekor requirement:
```bash
stella attest verify --artifact "$DIGEST" --skip-rekor
```

---

### Verification Failures

#### Error: "Certificate identity mismatch"

**Symptoms:**
```
Error: certificate identity 'repo:org/repo:ref:refs/heads/feature'
       does not match expected 'repo:org/repo:ref:refs/heads/main'
```

**Cause:** Artifact was signed from a different branch/ref than expected.

**Solutions:**

1. **Use regex for flexibility:**
```bash
stella attest verify \
  --artifact "$DIGEST" \
  --certificate-identity "repo:org/repo:.*" \
  --certificate-oidc-issuer "https://token.actions.githubusercontent.com"
```

2. **Verify expected signing context:**
```bash
# Check what identity was actually used
stella attest inspect --artifact "$DIGEST" --show-identity
```

#### Error: "Certificate OIDC issuer mismatch"

**Symptoms:**
```
Error: certificate issuer 'https://gitlab.com'
       does not match expected 'https://token.actions.githubusercontent.com'
```

**Cause:** Artifact was signed by a different CI platform.

**Solution:** Update verification to accept correct issuer:
```bash
# For GitLab-signed artifacts
stella attest verify \
  --artifact "$DIGEST" \
  --certificate-identity "project_path:org/repo:.*" \
  --certificate-oidc-issuer "https://gitlab.com"
```

#### Error: "Signature expired"

**Symptoms:**
```
Error: certificate validity period has expired
```

**Cause:** Fulcio certificates are short-lived (10 minutes). Verification after expiry requires Rekor proof.

**Solution:** Ensure Rekor verification is enabled:
```bash
stella attest verify \
  --artifact "$DIGEST" \
  --require-rekor \
  --certificate-identity "..." \
  --certificate-oidc-issuer "..."
```

---

### Platform-Specific Issues

#### GitHub Actions: "Resource not accessible by integration"

**Symptoms:**
```
Error: Resource not accessible by integration
```

**Cause:** GitHub App or token lacks required permissions.

**Solution:** Ensure workflow has correct permissions:
```yaml
permissions:
  id-token: write   # For OIDC token
  contents: read    # For checkout
  packages: write   # If pushing to GHCR
  attestations: write  # For GitHub attestations
```

#### GitLab CI: "id_tokens not available"

**Symptoms:**
```
Error: STELLAOPS_OIDC_TOKEN variable not set
```

**Cause:** GitLab version doesn't support `id_tokens` or feature is disabled.

**Solutions:**

1. Check GitLab version (requires 15.7+)
2. Enable CI/CD OIDC in project settings:
   - Settings > CI/CD > Token Access
   - Enable "Allow CI job tokens from the following projects"

3. Use service account as fallback:
```yaml
sign:
  script:
    - |
      if [ -z "$STELLAOPS_OIDC_TOKEN" ]; then
        # Fallback to service account
        stella attest sign --key "$SIGNING_KEY" --artifact "$DIGEST"
      else
        stella attest sign --keyless --artifact "$DIGEST"
      fi
```

#### Gitea: OIDC Token Format

**Symptoms:**
```
Error: Invalid OIDC token format
```

**Cause:** Gitea Actions uses different token acquisition method.

**Solution:**
```yaml
- name: Get OIDC Token
  run: |
    # Gitea provides token directly in environment
    if [ -n "$ACTIONS_ID_TOKEN" ]; then
      echo "token=$ACTIONS_ID_TOKEN" >> $GITHUB_OUTPUT
    else
      echo "::error::OIDC token not available"
      exit 1
    fi
```

---

### Network and Connectivity

#### Error: "Connection refused" to Sigstore services

**Symptoms:**
```
Error: dial tcp: connection refused
```

**Cause:** Firewall blocking outbound connections.

**Required endpoints:**
| Service | URL | Purpose |
|---------|-----|---------|
| Fulcio | `https://fulcio.sigstore.dev` | Certificate issuance |
| Rekor | `https://rekor.sigstore.dev` | Transparency log |
| TUF | `https://tuf-repo-cdn.sigstore.dev` | Trust root |
| OIDC | CI platform URL | Token validation |

**Solution:** Allow outbound HTTPS to these endpoints, or use self-hosted Sigstore.

#### Proxy Configuration

```yaml
- name: Sign with proxy
  env:
    HTTPS_PROXY: http://proxy.internal:8080
    NO_PROXY: internal.corp.com
    STELLAOPS_OIDC_TOKEN: ${{ steps.oidc.outputs.token }}
  run: stella attest sign --keyless --artifact "$DIGEST"
```

---

## Debugging Commands

### Inspect OIDC Token
```bash
# Decode token payload (never log in production)
echo $OIDC_TOKEN | cut -d. -f2 | base64 -d 2>/dev/null | jq .
```

### Verify Fulcio Connectivity
```bash
curl -v https://fulcio.sigstore.dev/api/v2/configuration
```

### Check Rekor Entry
```bash
# Search by artifact hash
rekor-cli search --sha "sha256:abc123..."

# Get entry details
rekor-cli get --uuid "24296fb24b8ad77a..."
```

### Inspect Attestation
```bash
stella attest inspect \
  --artifact "$DIGEST" \
  --show-certificate \
  --show-rekor-entry
```

### Verbose Signing
```bash
STELLAOPS_LOG_LEVEL=debug stella attest sign --keyless --artifact "$DIGEST"
```

---

## Getting Help

1. **Check service status:** https://status.sigstore.dev/
2. **StellaOps documentation:** https://docs.stella-ops.org/
3. **Sigstore documentation:** https://docs.sigstore.dev/
4. **File an issue:** https://github.com/stella-ops/stellaops/issues

When reporting issues, include:
- CI platform and version
- StellaOps CLI version (`stella --version`)
- Sanitized error output (remove tokens/secrets)
- Relevant workflow configuration