git.stella-ops.org/docs/modules/advisory-ai/llm-setup-guide.md

LLM Provider Setup Guide

This guide explains how to configure an LLM (Large Language Model) provider for AdvisoryAI features in StellaOps.

Overview

AdvisoryAI uses LLM providers to power AI-assisted vulnerability analysis, advisory recommendations, and conversational assistance. You can choose from several supported providers based on your requirements for privacy, performance, and cost.

Supported Providers

Provider	Description	Requirements
OpenAI	GPT-4o, GPT-4, GPT-3.5 Turbo	API key
Anthropic Claude	Claude 4 Sonnet, Claude 3.5 Sonnet, Claude 3 Opus	API key
Google Gemini	Gemini 1.5 Flash, Gemini 1.5 Pro	API key
Ollama	Local LLM (Llama 3, Mistral, etc.)	Local Ollama instance

Quick Start

Using the Setup Wizard (Recommended)

Run the interactive setup wizard to configure an LLM provider:

stella setup --step llm

The wizard will:

1. Present available provider options
2. Prompt for required credentials
3. Test API connectivity
4. Save the configuration

Using Environment Variables

You can also configure providers using environment variables:

# OpenAI
export OPENAI_API_KEY="sk-..."

# Anthropic Claude
export ANTHROPIC_API_KEY="sk-ant-..."

# Google Gemini
export GEMINI_API_KEY="AIza..."
# or
export GOOGLE_API_KEY="AIza..."

Provider Configuration

OpenAI

Configuration file: etc/llm-providers/openai.yaml

enabled: true
priority: 100

api:
  apiKey: "${OPENAI_API_KEY}"
  baseUrl: "https://api.openai.com/v1"

model:
  name: "gpt-4o"
  fallbacks:
    - "gpt-4-turbo"
    - "gpt-3.5-turbo"

inference:
  temperature: 0.0
  maxTokens: 8192
  seed: 42

Models available:

• gpt-4o - Recommended for most use cases
• gpt-4-turbo - High performance, higher cost
• gpt-4 - Previous generation
• gpt-3.5-turbo - Lower cost, faster

Anthropic Claude

Configuration file: etc/llm-providers/claude.yaml

enabled: true
priority: 100

api:
  apiKey: "${ANTHROPIC_API_KEY}"
  baseUrl: "https://api.anthropic.com"

model:
  name: "claude-sonnet-4-20250514"
  fallbacks:
    - "claude-3-5-sonnet-20241022"
    - "claude-3-haiku-20240307"

inference:
  temperature: 0.0
  maxTokens: 8192

Models available:

• claude-sonnet-4-20250514 - Latest Sonnet model (recommended)
• claude-3-5-sonnet-20241022 - Claude 3.5 Sonnet
• claude-3-opus-20240229 - Highest capability
• claude-3-haiku-20240307 - Fastest, lowest cost

Google Gemini

Configuration file: etc/llm-providers/gemini.yaml

enabled: true
priority: 100

api:
  apiKey: "${GEMINI_API_KEY}"
  baseUrl: "https://generativelanguage.googleapis.com/v1beta"

model:
  name: "gemini-1.5-flash"
  fallbacks:
    - "gemini-1.5-pro"
    - "gemini-1.0-pro"

inference:
  temperature: 0.0
  maxTokens: 8192
  topP: 1.0
  topK: 40

Models available:

• gemini-1.5-flash - Fast, cost-effective (recommended)
• gemini-1.5-pro - Higher capability
• gemini-1.0-pro - Previous generation

Ollama (Local)

Configuration file: etc/llm-providers/ollama.yaml

enabled: true
priority: 50

api:
  endpoint: "http://localhost:11434"

model:
  name: "llama3:8b"
  fallbacks:
    - "mistral:7b"

inference:
  temperature: 0.0
  maxTokens: 4096

Prerequisites:

1. Install Ollama: https://ollama.ai
2. Pull a model: ollama pull llama3:8b
3. Start Ollama: ollama serve

Recommended models:

• llama3:8b - Good balance of speed and capability
• llama3:70b - Higher capability, requires more resources
• mistral:7b - Fast, efficient
• codellama:7b - Optimized for code

Checking Configuration

Using Doctor

Run the Doctor checks to validate your LLM configuration:

# Check all AI-related configuration
stella doctor run --category ai

# Check specific provider
stella doctor run --check check.ai.provider.openai
stella doctor run --check check.ai.provider.claude
stella doctor run --check check.ai.provider.gemini

Using the CLI

Check your AdvisoryAI chat configuration:

stella advise chat-doctor

Troubleshooting

"AI/LLM provider not configured"

This error appears when no LLM provider is configured. Solutions:

1. Run stella setup --step llm to configure a provider
2. Set environment variables for your preferred provider
3. Create a configuration file in etc/llm-providers/

API Key Invalid

If you receive authentication errors:

1. Verify your API key is correct
2. Check the API key has not expired
3. Ensure billing is active on your provider account
4. For Gemini, ensure the Generative Language API is enabled

Connection Timeout

If connections time out:

1. Check network connectivity to the provider endpoint
2. Verify proxy settings if behind a firewall
3. For Ollama, ensure the service is running locally

Rate Limiting

If you encounter rate limits:

1. Reduce request frequency
2. Consider upgrading your API tier
3. Enable request queueing in configuration

Offline/Air-Gapped Operation

For air-gapped deployments, use Ollama with locally-available models:

1. Download models on a connected system
2. Transfer model files to the air-gapped environment
3. Configure Ollama with local models
4. Set AdvisoryAI:DefaultProvider to ollama

Security Considerations

1. API Key Storage: Never commit API keys to version control. Use environment variables or secure vaults.
2. Data Privacy: Be aware of data sent to cloud providers. Use Ollama for sensitive data.
3. Rate Limiting: Configure appropriate rate limits to prevent abuse.
4. Audit Logging: Enable audit logging for all LLM interactions.

Related Documentation

• AdvisoryAI Architecture
• Chat Interface
• Deployment Guide
• Assistant Guardrails