git.stella-ops.org/docs/modules/advisory-ai/llm-setup-guide.md

# LLM Provider Setup Guide

This guide explains how to configure an LLM (Large Language Model) provider for AdvisoryAI features in StellaOps.

## Overview

AdvisoryAI uses LLM providers to power AI-assisted vulnerability analysis, advisory recommendations, and conversational assistance. You can choose from several supported providers based on your requirements for privacy, performance, and cost.

## Supported Providers

| Provider | Description | Requirements |
|----------|-------------|--------------|
| **OpenAI** | GPT-4o, GPT-4, GPT-3.5 Turbo | API key |
| **Anthropic Claude** | Claude 4 Sonnet, Claude 3.5 Sonnet, Claude 3 Opus | API key |
| **Google Gemini** | Gemini 1.5 Flash, Gemini 1.5 Pro | API key |
| **Ollama** | Local LLM (Llama 3, Mistral, etc.) | Local Ollama instance |

## Quick Start

### Using the Setup Wizard (Recommended)

Run the interactive setup wizard to configure an LLM provider:

```bash
stella setup --step llm
```

The wizard will:
1. Present available provider options
2. Prompt for required credentials
3. Test API connectivity
4. Save the configuration

### Using Environment Variables

You can also configure providers using environment variables:

```bash
# OpenAI
export OPENAI_API_KEY="sk-..."

# Anthropic Claude
export ANTHROPIC_API_KEY="sk-ant-..."

# Google Gemini
export GEMINI_API_KEY="AIza..."
# or
export GOOGLE_API_KEY="AIza..."
```

## Provider Configuration

### OpenAI

**Configuration file:** `etc/llm-providers/openai.yaml`

```yaml
enabled: true
priority: 100

api:
  apiKey: "${OPENAI_API_KEY}"
  baseUrl: "https://api.openai.com/v1"

model:
  name: "gpt-4o"
  fallbacks:
    - "gpt-4-turbo"
    - "gpt-3.5-turbo"

inference:
  temperature: 0.0
  maxTokens: 8192
  seed: 42
```

**Models available:**
- `gpt-4o` - Recommended for most use cases
- `gpt-4-turbo` - High performance, higher cost
- `gpt-4` - Previous generation
- `gpt-3.5-turbo` - Lower cost, faster

### Anthropic Claude

**Configuration file:** `etc/llm-providers/claude.yaml`

```yaml
enabled: true
priority: 100

api:
  apiKey: "${ANTHROPIC_API_KEY}"
  baseUrl: "https://api.anthropic.com"

model:
  name: "claude-sonnet-4-20250514"
  fallbacks:
    - "claude-3-5-sonnet-20241022"
    - "claude-3-haiku-20240307"

inference:
  temperature: 0.0
  maxTokens: 8192
```

**Models available:**
- `claude-sonnet-4-20250514` - Latest Sonnet model (recommended)
- `claude-3-5-sonnet-20241022` - Claude 3.5 Sonnet
- `claude-3-opus-20240229` - Highest capability
- `claude-3-haiku-20240307` - Fastest, lowest cost

### Google Gemini

**Configuration file:** `etc/llm-providers/gemini.yaml`

```yaml
enabled: true
priority: 100

api:
  apiKey: "${GEMINI_API_KEY}"
  baseUrl: "https://generativelanguage.googleapis.com/v1beta"

model:
  name: "gemini-1.5-flash"
  fallbacks:
    - "gemini-1.5-pro"
    - "gemini-1.0-pro"

inference:
  temperature: 0.0
  maxTokens: 8192
  topP: 1.0
  topK: 40
```

**Models available:**
- `gemini-1.5-flash` - Fast, cost-effective (recommended)
- `gemini-1.5-pro` - Higher capability
- `gemini-1.0-pro` - Previous generation

### Ollama (Local)

**Configuration file:** `etc/llm-providers/ollama.yaml`

```yaml
enabled: true
priority: 50

api:
  endpoint: "http://localhost:11434"

model:
  name: "llama3:8b"
  fallbacks:
    - "mistral:7b"

inference:
  temperature: 0.0
  maxTokens: 4096
```

**Prerequisites:**
1. Install Ollama: https://ollama.ai
2. Pull a model: `ollama pull llama3:8b`
3. Start Ollama: `ollama serve`

**Recommended models:**
- `llama3:8b` - Good balance of speed and capability
- `llama3:70b` - Higher capability, requires more resources
- `mistral:7b` - Fast, efficient
- `codellama:7b` - Optimized for code

## Checking Configuration

### Using Doctor

Run the Doctor checks to validate your LLM configuration:

```bash
# Check all AI-related configuration
stella doctor run --category ai

# Check specific provider
stella doctor run --check check.ai.provider.openai
stella doctor run --check check.ai.provider.claude
stella doctor run --check check.ai.provider.gemini
```

### Using the CLI

Check your AdvisoryAI chat configuration:

```bash
stella advise chat-doctor
```

## Troubleshooting

### "AI/LLM provider not configured"

This error appears when no LLM provider is configured. Solutions:

1. Run `stella setup --step llm` to configure a provider
2. Set environment variables for your preferred provider
3. Create a configuration file in `etc/llm-providers/`

### API Key Invalid

If you receive authentication errors:

1. Verify your API key is correct
2. Check the API key has not expired
3. Ensure billing is active on your provider account
4. For Gemini, ensure the Generative Language API is enabled

### Connection Timeout

If connections time out:

1. Check network connectivity to the provider endpoint
2. Verify proxy settings if behind a firewall
3. For Ollama, ensure the service is running locally

### Rate Limiting

If you encounter rate limits:

1. Reduce request frequency
2. Consider upgrading your API tier
3. Enable request queueing in configuration

## Offline/Air-Gapped Operation

For air-gapped deployments, use Ollama with locally-available models:

1. Download models on a connected system
2. Transfer model files to the air-gapped environment
3. Configure Ollama with local models
4. Set `AdvisoryAI:DefaultProvider` to `ollama`

## Security Considerations

1. **API Key Storage:** Never commit API keys to version control. Use environment variables or secure vaults.
2. **Data Privacy:** Be aware of data sent to cloud providers. Use Ollama for sensitive data.
3. **Rate Limiting:** Configure appropriate rate limits to prevent abuse.
4. **Audit Logging:** Enable audit logging for all LLM interactions.

## Related Documentation

- [AdvisoryAI Architecture](./architecture.md)
- [Chat Interface](./chat-interface.md)
- [Deployment Guide](./deployment.md)
- [Assistant Guardrails](/docs/security/assistant-guardrails.md)