Add property-based tests for SBOM/VEX document ordering and Unicode normalization determinism
- Implement `SbomVexOrderingDeterminismProperties` for testing component list and vulnerability metadata hash consistency. - Create `UnicodeNormalizationDeterminismProperties` to validate NFC normalization and Unicode string handling. - Add project file for `StellaOps.Testing.Determinism.Properties` with necessary dependencies. - Introduce CI/CD template validation tests including YAML syntax checks and documentation content verification. - Create validation script for CI/CD templates ensuring all required files and structures are present.
This commit is contained in:
399
docs/guides/keyless-signing-troubleshooting.md
Normal file
399
docs/guides/keyless-signing-troubleshooting.md
Normal file
@@ -0,0 +1,399 @@
|
||||
# Keyless Signing Troubleshooting Guide
|
||||
|
||||
This guide covers common issues when integrating StellaOps keyless signing into CI/CD pipelines.
|
||||
|
||||
## Common Errors
|
||||
|
||||
### OIDC Token Acquisition Failures
|
||||
|
||||
#### Error: "Unable to get OIDC token"
|
||||
|
||||
**Symptoms:**
|
||||
```
|
||||
Error: Unable to get ACTIONS_ID_TOKEN_REQUEST_URL
|
||||
```
|
||||
|
||||
**Cause:** The workflow doesn't have `id-token: write` permission.
|
||||
|
||||
**Solution:**
|
||||
```yaml
|
||||
# GitHub Actions
|
||||
permissions:
|
||||
id-token: write
|
||||
contents: read
|
||||
|
||||
# GitLab CI
|
||||
job:
|
||||
id_tokens:
|
||||
STELLAOPS_OIDC_TOKEN:
|
||||
aud: sigstore
|
||||
```
|
||||
|
||||
#### Error: "Token audience mismatch"
|
||||
|
||||
**Symptoms:**
|
||||
```
|
||||
Error: Token audience 'api://default' does not match expected 'sigstore'
|
||||
```
|
||||
|
||||
**Cause:** OIDC token was requested with wrong audience.
|
||||
|
||||
**Solution:**
|
||||
```yaml
|
||||
# GitHub Actions
|
||||
OIDC_TOKEN=$(curl -sLS "${ACTIONS_ID_TOKEN_REQUEST_URL}&audience=sigstore" \
|
||||
-H "Authorization: bearer ${ACTIONS_ID_TOKEN_REQUEST_TOKEN}")
|
||||
|
||||
# GitLab CI
|
||||
id_tokens:
|
||||
STELLAOPS_OIDC_TOKEN:
|
||||
aud: sigstore # Must be 'sigstore' for Fulcio
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Fulcio Certificate Errors
|
||||
|
||||
#### Error: "Failed to get certificate from Fulcio"
|
||||
|
||||
**Symptoms:**
|
||||
```
|
||||
Error: error getting certificate from Fulcio: 401 Unauthorized
|
||||
```
|
||||
|
||||
**Causes:**
|
||||
1. OIDC token expired (tokens are short-lived, typically 5-10 minutes)
|
||||
2. Fulcio doesn't recognize the OIDC issuer
|
||||
3. Network connectivity issues to Fulcio
|
||||
|
||||
**Solutions:**
|
||||
|
||||
1. **Token expiry:** Request token immediately before signing:
|
||||
```yaml
|
||||
- name: Get OIDC Token
|
||||
id: oidc
|
||||
run: |
|
||||
# Get fresh token right before signing
|
||||
OIDC_TOKEN=$(curl -sLS "${ACTIONS_ID_TOKEN_REQUEST_URL}&audience=sigstore" \
|
||||
-H "Authorization: bearer ${ACTIONS_ID_TOKEN_REQUEST_TOKEN}" \
|
||||
| jq -r '.value')
|
||||
echo "token=${OIDC_TOKEN}" >> $GITHUB_OUTPUT
|
||||
|
||||
- name: Sign (immediately after)
|
||||
env:
|
||||
STELLAOPS_OIDC_TOKEN: ${{ steps.oidc.outputs.token }}
|
||||
run: stella attest sign --keyless --artifact "$DIGEST"
|
||||
```
|
||||
|
||||
2. **Unknown issuer:** Ensure your CI platform is supported:
|
||||
- GitHub Actions: `https://token.actions.githubusercontent.com`
|
||||
- GitLab.com: `https://gitlab.com`
|
||||
- Self-hosted GitLab: Must be configured in Fulcio
|
||||
|
||||
3. **Network issues:** Check connectivity:
|
||||
```bash
|
||||
curl -v https://fulcio.sigstore.dev/api/v2/signingCert
|
||||
```
|
||||
|
||||
#### Error: "Certificate identity not found in token"
|
||||
|
||||
**Symptoms:**
|
||||
```
|
||||
Error: no matching subject or SAN found in OIDC token
|
||||
```
|
||||
|
||||
**Cause:** Token claims don't include expected identity fields.
|
||||
|
||||
**Solution:** Verify token contents:
|
||||
```bash
|
||||
# Decode and inspect token (don't do this in production logs)
|
||||
echo $OIDC_TOKEN | cut -d. -f2 | base64 -d | jq .
|
||||
```
|
||||
|
||||
Expected claims for GitHub Actions:
|
||||
```json
|
||||
{
|
||||
"sub": "repo:org/repo:ref:refs/heads/main",
|
||||
"iss": "https://token.actions.githubusercontent.com",
|
||||
"repository": "org/repo",
|
||||
"ref": "refs/heads/main"
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Rekor Transparency Log Errors
|
||||
|
||||
#### Error: "Failed to upload to Rekor"
|
||||
|
||||
**Symptoms:**
|
||||
```
|
||||
Error: error uploading entry to Rekor: 500 Internal Server Error
|
||||
```
|
||||
|
||||
**Causes:**
|
||||
1. Rekor service temporarily unavailable
|
||||
2. Entry too large
|
||||
3. Network issues
|
||||
|
||||
**Solutions:**
|
||||
|
||||
1. **Retry with backoff:**
|
||||
```yaml
|
||||
- name: Sign with retry
|
||||
run: |
|
||||
for i in 1 2 3; do
|
||||
stella attest sign --keyless --artifact "$DIGEST" && break
|
||||
echo "Attempt $i failed, retrying in 30s..."
|
||||
sleep 30
|
||||
done
|
||||
```
|
||||
|
||||
2. **Check Rekor status:** https://status.sigstore.dev/
|
||||
|
||||
3. **Use offline bundle (air-gapped):**
|
||||
```bash
|
||||
stella attest sign --keyless --artifact "$DIGEST" --offline-bundle
|
||||
```
|
||||
|
||||
#### Error: "Rekor entry not found"
|
||||
|
||||
**Symptoms:**
|
||||
```
|
||||
Error: entry not found in transparency log
|
||||
```
|
||||
|
||||
**Cause:** Verification requiring Rekor but entry wasn't logged (offline signing).
|
||||
|
||||
**Solution:** Either:
|
||||
- Sign with Rekor enabled (default)
|
||||
- Verify without Rekor requirement:
|
||||
```bash
|
||||
stella attest verify --artifact "$DIGEST" --skip-rekor
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Verification Failures
|
||||
|
||||
#### Error: "Certificate identity mismatch"
|
||||
|
||||
**Symptoms:**
|
||||
```
|
||||
Error: certificate identity 'repo:org/repo:ref:refs/heads/feature'
|
||||
does not match expected 'repo:org/repo:ref:refs/heads/main'
|
||||
```
|
||||
|
||||
**Cause:** Artifact was signed from a different branch/ref than expected.
|
||||
|
||||
**Solutions:**
|
||||
|
||||
1. **Use regex for flexibility:**
|
||||
```bash
|
||||
stella attest verify \
|
||||
--artifact "$DIGEST" \
|
||||
--certificate-identity "repo:org/repo:.*" \
|
||||
--certificate-oidc-issuer "https://token.actions.githubusercontent.com"
|
||||
```
|
||||
|
||||
2. **Verify expected signing context:**
|
||||
```bash
|
||||
# Check what identity was actually used
|
||||
stella attest inspect --artifact "$DIGEST" --show-identity
|
||||
```
|
||||
|
||||
#### Error: "Certificate OIDC issuer mismatch"
|
||||
|
||||
**Symptoms:**
|
||||
```
|
||||
Error: certificate issuer 'https://gitlab.com'
|
||||
does not match expected 'https://token.actions.githubusercontent.com'
|
||||
```
|
||||
|
||||
**Cause:** Artifact was signed by a different CI platform.
|
||||
|
||||
**Solution:** Update verification to accept correct issuer:
|
||||
```bash
|
||||
# For GitLab-signed artifacts
|
||||
stella attest verify \
|
||||
--artifact "$DIGEST" \
|
||||
--certificate-identity "project_path:org/repo:.*" \
|
||||
--certificate-oidc-issuer "https://gitlab.com"
|
||||
```
|
||||
|
||||
#### Error: "Signature expired"
|
||||
|
||||
**Symptoms:**
|
||||
```
|
||||
Error: certificate validity period has expired
|
||||
```
|
||||
|
||||
**Cause:** Fulcio certificates are short-lived (10 minutes). Verification after expiry requires Rekor proof.
|
||||
|
||||
**Solution:** Ensure Rekor verification is enabled:
|
||||
```bash
|
||||
stella attest verify \
|
||||
--artifact "$DIGEST" \
|
||||
--require-rekor \
|
||||
--certificate-identity "..." \
|
||||
--certificate-oidc-issuer "..."
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Platform-Specific Issues
|
||||
|
||||
#### GitHub Actions: "Resource not accessible by integration"
|
||||
|
||||
**Symptoms:**
|
||||
```
|
||||
Error: Resource not accessible by integration
|
||||
```
|
||||
|
||||
**Cause:** GitHub App or token lacks required permissions.
|
||||
|
||||
**Solution:** Ensure workflow has correct permissions:
|
||||
```yaml
|
||||
permissions:
|
||||
id-token: write # For OIDC token
|
||||
contents: read # For checkout
|
||||
packages: write # If pushing to GHCR
|
||||
attestations: write # For GitHub attestations
|
||||
```
|
||||
|
||||
#### GitLab CI: "id_tokens not available"
|
||||
|
||||
**Symptoms:**
|
||||
```
|
||||
Error: STELLAOPS_OIDC_TOKEN variable not set
|
||||
```
|
||||
|
||||
**Cause:** GitLab version doesn't support `id_tokens` or feature is disabled.
|
||||
|
||||
**Solutions:**
|
||||
|
||||
1. Check GitLab version (requires 15.7+)
|
||||
2. Enable CI/CD OIDC in project settings:
|
||||
- Settings > CI/CD > Token Access
|
||||
- Enable "Allow CI job tokens from the following projects"
|
||||
|
||||
3. Use service account as fallback:
|
||||
```yaml
|
||||
sign:
|
||||
script:
|
||||
- |
|
||||
if [ -z "$STELLAOPS_OIDC_TOKEN" ]; then
|
||||
# Fallback to service account
|
||||
stella attest sign --key "$SIGNING_KEY" --artifact "$DIGEST"
|
||||
else
|
||||
stella attest sign --keyless --artifact "$DIGEST"
|
||||
fi
|
||||
```
|
||||
|
||||
#### Gitea: OIDC Token Format
|
||||
|
||||
**Symptoms:**
|
||||
```
|
||||
Error: Invalid OIDC token format
|
||||
```
|
||||
|
||||
**Cause:** Gitea Actions uses different token acquisition method.
|
||||
|
||||
**Solution:**
|
||||
```yaml
|
||||
- name: Get OIDC Token
|
||||
run: |
|
||||
# Gitea provides token directly in environment
|
||||
if [ -n "$ACTIONS_ID_TOKEN" ]; then
|
||||
echo "token=$ACTIONS_ID_TOKEN" >> $GITHUB_OUTPUT
|
||||
else
|
||||
echo "::error::OIDC token not available"
|
||||
exit 1
|
||||
fi
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Network and Connectivity
|
||||
|
||||
#### Error: "Connection refused" to Sigstore services
|
||||
|
||||
**Symptoms:**
|
||||
```
|
||||
Error: dial tcp: connection refused
|
||||
```
|
||||
|
||||
**Cause:** Firewall blocking outbound connections.
|
||||
|
||||
**Required endpoints:**
|
||||
| Service | URL | Purpose |
|
||||
|---------|-----|---------|
|
||||
| Fulcio | `https://fulcio.sigstore.dev` | Certificate issuance |
|
||||
| Rekor | `https://rekor.sigstore.dev` | Transparency log |
|
||||
| TUF | `https://tuf-repo-cdn.sigstore.dev` | Trust root |
|
||||
| OIDC | CI platform URL | Token validation |
|
||||
|
||||
**Solution:** Allow outbound HTTPS to these endpoints, or use self-hosted Sigstore.
|
||||
|
||||
#### Proxy Configuration
|
||||
|
||||
```yaml
|
||||
- name: Sign with proxy
|
||||
env:
|
||||
HTTPS_PROXY: http://proxy.internal:8080
|
||||
NO_PROXY: internal.corp.com
|
||||
STELLAOPS_OIDC_TOKEN: ${{ steps.oidc.outputs.token }}
|
||||
run: stella attest sign --keyless --artifact "$DIGEST"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Debugging Commands
|
||||
|
||||
### Inspect OIDC Token
|
||||
```bash
|
||||
# Decode token payload (never log in production)
|
||||
echo $OIDC_TOKEN | cut -d. -f2 | base64 -d 2>/dev/null | jq .
|
||||
```
|
||||
|
||||
### Verify Fulcio Connectivity
|
||||
```bash
|
||||
curl -v https://fulcio.sigstore.dev/api/v2/configuration
|
||||
```
|
||||
|
||||
### Check Rekor Entry
|
||||
```bash
|
||||
# Search by artifact hash
|
||||
rekor-cli search --sha "sha256:abc123..."
|
||||
|
||||
# Get entry details
|
||||
rekor-cli get --uuid "24296fb24b8ad77a..."
|
||||
```
|
||||
|
||||
### Inspect Attestation
|
||||
```bash
|
||||
stella attest inspect \
|
||||
--artifact "$DIGEST" \
|
||||
--show-certificate \
|
||||
--show-rekor-entry
|
||||
```
|
||||
|
||||
### Verbose Signing
|
||||
```bash
|
||||
STELLAOPS_LOG_LEVEL=debug stella attest sign --keyless --artifact "$DIGEST"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Getting Help
|
||||
|
||||
1. **Check service status:** https://status.sigstore.dev/
|
||||
2. **StellaOps documentation:** https://docs.stella-ops.org/
|
||||
3. **Sigstore documentation:** https://docs.sigstore.dev/
|
||||
4. **File an issue:** https://github.com/stella-ops/stellaops/issues
|
||||
|
||||
When reporting issues, include:
|
||||
- CI platform and version
|
||||
- StellaOps CLI version (`stella --version`)
|
||||
- Sanitized error output (remove tokens/secrets)
|
||||
- Relevant workflow configuration
|
||||
Reference in New Issue
Block a user