audit notes work completed, test fixes work (95% done), new sprints, new data sources setup and configuration
This commit is contained in:
master
254
docs/modules/advisory-ai/llm-setup-guide.md
Normal file
254
docs/modules/advisory-ai/llm-setup-guide.md
Normal file
@@ -0,0 +1,254 @@
|
||||
# LLM Provider Setup Guide
|
||||
|
||||
This guide explains how to configure an LLM (Large Language Model) provider for AdvisoryAI features in StellaOps.
|
||||
|
||||
## Overview
|
||||
|
||||
AdvisoryAI uses LLM providers to power AI-assisted vulnerability analysis, advisory recommendations, and conversational assistance. You can choose from several supported providers based on your requirements for privacy, performance, and cost.
|
||||
|
||||
## Supported Providers
|
||||
|
||||
| Provider | Description | Requirements |
|
||||
|----------|-------------|--------------|
|
||||
| **OpenAI** | GPT-4o, GPT-4, GPT-3.5 Turbo | API key |
|
||||
| **Anthropic Claude** | Claude 4 Sonnet, Claude 3.5 Sonnet, Claude 3 Opus | API key |
|
||||
| **Google Gemini** | Gemini 1.5 Flash, Gemini 1.5 Pro | API key |
|
||||
| **Ollama** | Local LLM (Llama 3, Mistral, etc.) | Local Ollama instance |
|
||||
|
||||
## Quick Start
|
||||
|
||||
### Using the Setup Wizard (Recommended)
|
||||
|
||||
Run the interactive setup wizard to configure an LLM provider:
|
||||
|
||||
```bash
|
||||
stella setup --step llm
|
||||
```
|
||||
|
||||
The wizard will:
|
||||
1. Present available provider options
|
||||
2. Prompt for required credentials
|
||||
3. Test API connectivity
|
||||
4. Save the configuration
|
||||
|
||||
### Using Environment Variables
|
||||
|
||||
You can also configure providers using environment variables:
|
||||
|
||||
```bash
|
||||
# OpenAI
|
||||
export OPENAI_API_KEY="sk-..."
|
||||
|
||||
# Anthropic Claude
|
||||
export ANTHROPIC_API_KEY="sk-ant-..."
|
||||
|
||||
# Google Gemini
|
||||
export GEMINI_API_KEY="AIza..."
|
||||
# or
|
||||
export GOOGLE_API_KEY="AIza..."
|
||||
```
|
||||
|
||||
## Provider Configuration
|
||||
|
||||
### OpenAI
|
||||
|
||||
**Configuration file:** `etc/llm-providers/openai.yaml`
|
||||
|
||||
```yaml
|
||||
enabled: true
|
||||
priority: 100
|
||||
|
||||
api:
|
||||
apiKey: "${OPENAI_API_KEY}"
|
||||
baseUrl: "https://api.openai.com/v1"
|
||||
|
||||
model:
|
||||
name: "gpt-4o"
|
||||
fallbacks:
|
||||
- "gpt-4-turbo"
|
||||
- "gpt-3.5-turbo"
|
||||
|
||||
inference:
|
||||
temperature: 0.0
|
||||
maxTokens: 8192
|
||||
seed: 42
|
||||
```
|
||||
|
||||
**Models available:**
|
||||
- `gpt-4o` - Recommended for most use cases
|
||||
- `gpt-4-turbo` - High performance, higher cost
|
||||
- `gpt-4` - Previous generation
|
||||
- `gpt-3.5-turbo` - Lower cost, faster
|
||||
|
||||
### Anthropic Claude
|
||||
|
||||
**Configuration file:** `etc/llm-providers/claude.yaml`
|
||||
|
||||
```yaml
|
||||
enabled: true
|
||||
priority: 100
|
||||
|
||||
api:
|
||||
apiKey: "${ANTHROPIC_API_KEY}"
|
||||
baseUrl: "https://api.anthropic.com"
|
||||
|
||||
model:
|
||||
name: "claude-sonnet-4-20250514"
|
||||
fallbacks:
|
||||
- "claude-3-5-sonnet-20241022"
|
||||
- "claude-3-haiku-20240307"
|
||||
|
||||
inference:
|
||||
temperature: 0.0
|
||||
maxTokens: 8192
|
||||
```
|
||||
|
||||
**Models available:**
|
||||
- `claude-sonnet-4-20250514` - Latest Sonnet model (recommended)
|
||||
- `claude-3-5-sonnet-20241022` - Claude 3.5 Sonnet
|
||||
- `claude-3-opus-20240229` - Highest capability
|
||||
- `claude-3-haiku-20240307` - Fastest, lowest cost
|
||||
|
||||
### Google Gemini
|
||||
|
||||
**Configuration file:** `etc/llm-providers/gemini.yaml`
|
||||
|
||||
```yaml
|
||||
enabled: true
|
||||
priority: 100
|
||||
|
||||
api:
|
||||
apiKey: "${GEMINI_API_KEY}"
|
||||
baseUrl: "https://generativelanguage.googleapis.com/v1beta"
|
||||
|
||||
model:
|
||||
name: "gemini-1.5-flash"
|
||||
fallbacks:
|
||||
- "gemini-1.5-pro"
|
||||
- "gemini-1.0-pro"
|
||||
|
||||
inference:
|
||||
temperature: 0.0
|
||||
maxTokens: 8192
|
||||
topP: 1.0
|
||||
topK: 40
|
||||
```
|
||||
|
||||
**Models available:**
|
||||
- `gemini-1.5-flash` - Fast, cost-effective (recommended)
|
||||
- `gemini-1.5-pro` - Higher capability
|
||||
- `gemini-1.0-pro` - Previous generation
|
||||
|
||||
### Ollama (Local)
|
||||
|
||||
**Configuration file:** `etc/llm-providers/ollama.yaml`
|
||||
|
||||
```yaml
|
||||
enabled: true
|
||||
priority: 50
|
||||
|
||||
api:
|
||||
endpoint: "http://localhost:11434"
|
||||
|
||||
model:
|
||||
name: "llama3:8b"
|
||||
fallbacks:
|
||||
- "mistral:7b"
|
||||
|
||||
inference:
|
||||
temperature: 0.0
|
||||
maxTokens: 4096
|
||||
```
|
||||
|
||||
**Prerequisites:**
|
||||
1. Install Ollama: https://ollama.ai
|
||||
2. Pull a model: `ollama pull llama3:8b`
|
||||
3. Start Ollama: `ollama serve`
|
||||
|
||||
**Recommended models:**
|
||||
- `llama3:8b` - Good balance of speed and capability
|
||||
- `llama3:70b` - Higher capability, requires more resources
|
||||
- `mistral:7b` - Fast, efficient
|
||||
- `codellama:7b` - Optimized for code
|
||||
|
||||
## Checking Configuration
|
||||
|
||||
### Using Doctor
|
||||
|
||||
Run the Doctor checks to validate your LLM configuration:
|
||||
|
||||
```bash
|
||||
# Check all AI-related configuration
|
||||
stella doctor run --category ai
|
||||
|
||||
# Check specific provider
|
||||
stella doctor run --check check.ai.provider.openai
|
||||
stella doctor run --check check.ai.provider.claude
|
||||
stella doctor run --check check.ai.provider.gemini
|
||||
```
|
||||
|
||||
### Using the CLI
|
||||
|
||||
Check your AdvisoryAI chat configuration:
|
||||
|
||||
```bash
|
||||
stella advise chat-doctor
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### "AI/LLM provider not configured"
|
||||
|
||||
This error appears when no LLM provider is configured. Solutions:
|
||||
|
||||
1. Run `stella setup --step llm` to configure a provider
|
||||
2. Set environment variables for your preferred provider
|
||||
3. Create a configuration file in `etc/llm-providers/`
|
||||
|
||||
### API Key Invalid
|
||||
|
||||
If you receive authentication errors:
|
||||
|
||||
1. Verify your API key is correct
|
||||
2. Check the API key has not expired
|
||||
3. Ensure billing is active on your provider account
|
||||
4. For Gemini, ensure the Generative Language API is enabled
|
||||
|
||||
### Connection Timeout
|
||||
|
||||
If connections time out:
|
||||
|
||||
1. Check network connectivity to the provider endpoint
|
||||
2. Verify proxy settings if behind a firewall
|
||||
3. For Ollama, ensure the service is running locally
|
||||
|
||||
### Rate Limiting
|
||||
|
||||
If you encounter rate limits:
|
||||
|
||||
1. Reduce request frequency
|
||||
2. Consider upgrading your API tier
|
||||
3. Enable request queueing in configuration
|
||||
|
||||
## Offline/Air-Gapped Operation
|
||||
|
||||
For air-gapped deployments, use Ollama with locally-available models:
|
||||
|
||||
1. Download models on a connected system
|
||||
2. Transfer model files to the air-gapped environment
|
||||
3. Configure Ollama with local models
|
||||
4. Set `AdvisoryAI:DefaultProvider` to `ollama`
|
||||
|
||||
## Security Considerations
|
||||
|
||||
1. **API Key Storage:** Never commit API keys to version control. Use environment variables or secure vaults.
|
||||
2. **Data Privacy:** Be aware of data sent to cloud providers. Use Ollama for sensitive data.
|
||||
3. **Rate Limiting:** Configure appropriate rate limits to prevent abuse.
|
||||
4. **Audit Logging:** Enable audit logging for all LLM interactions.
|
||||
|
||||
## Related Documentation
|
||||
|
||||
- [AdvisoryAI Architecture](./architecture.md)
|
||||
- [Chat Interface](./chat-interface.md)
|
||||
- [Deployment Guide](./deployment.md)
|
||||
- [Assistant Guardrails](/docs/security/assistant-guardrails.md)
|
||||
Reference in New Issue
Block a user