Skip to main content
LiteLLM provides a unified interface to multiple model providers, allowing you to use any model with an OpenAI-compatible interface while keeping your API keys private and centralized. This guide shows you how to set up and configure LiteLLM for use with qckfx.

​
Why Use LiteLLM?

LiteLLM offers several advantages for agent development:
  • Unified Interface: Use any model provider through a single OpenAI-compatible API
  • Centralized API Keys: Store all your API keys in one secure location
  • Model Abstraction: Switch between models without changing your code
  • Rate Limiting: Built-in rate limiting and request management
  • Logging: Comprehensive request and response logging
  • Cost Tracking: Monitor usage and costs across providers

​
Quick Start with Docker

The qckfx agent repository includes a pre-configured LiteLLM setup:

​
1. Navigate to LiteLLM Directory

cd path/to/agent-core/litellm

​
2. Build the Docker Image

docker build -t litellm .

​
3. Run LiteLLM with Your API Keys

docker run -p 8001:8001 \
  -e OPENAI_API_KEY=your_openai_key \
  -e ANTHROPIC_API_KEY=your_anthropic_key \
  -e GEMINI_API_KEY=your_gemini_key \
  litellm

​
4. Configure qckfx to Use LiteLLM

# Set the base URL to your LiteLLM instance
export LLM_BASE_URL=http://localhost:8001

# Use any model from the LiteLLM configuration
qckfx -m claude-sonnet-4 "Hello, world!"
The LiteLLM proxy will be available at http://localhost:8001 and provides an OpenAI-compatible API endpoint.

​
Pre-configured Models

The included LiteLLM configuration supports these models out of the box:

​
Anthropic Models

Model NameProvider ModelDescription
claude-sonnet-3-7anthropic/claude-3-7-sonnet-20250219Latest Claude 3.7 Sonnet
claude-sonnet-4anthropic/claude-sonnet-4-20250514Claude 4 Sonnet

​
OpenAI Models

Model NameProvider ModelDescription
o3openai/o3OpenAI o3 model
gpt-o4-miniopenai/o4-miniGPT-4 Mini variant
gpt-4.1-miniopenai/gpt-4.1-miniGPT-4.1 Mini
gpt-4.1-nanoopenai/gpt-4.1-nanoGPT-4.1 Nano

​
Google Models

Model NameProvider ModelDescription
gemini-2.5-progemini/gemini-2.5-pro-preview-03-25Gemini 2.5 Pro

​
Configuration File

The LiteLLM configuration is defined in config.yaml: 
model_list:
  - model_name: claude-sonnet-4
    litellm_params:
      model: anthropic/claude-sonnet-4-20250514
      api_base: https://api.anthropic.com
      api_key: 'os.environ/ANTHROPIC_API_KEY'
      drop_params: true
  
  - model_name: gemini-2.5-pro
    litellm_params:
      model: gemini/gemini-2.5-pro-preview-03-25
      api_key: 'os.environ/GEMINI_API_KEY'
      drop_params: true
  
  # ... more models

​
Configuration Options

FieldDescription
model_nameFriendly name used in qckfx commands
modelProvider-specific model identifier
api_baseProvider API endpoint (optional)
api_keyEnvironment variable containing API key
drop_paramsDrop unsupported parameters for compatibility

​
Adding New Models

To add support for additional models, edit the config.yaml file: 
model_list:
  # ... existing models
  
  - model_name: my-custom-model
    litellm_params:
      model: provider/model-name
      api_key: 'os.environ/MY_PROVIDER_API_KEY'
      drop_params: true
Refer to the LiteLLM documentation for provider-specific configuration options.

​
Environment Variables

LiteLLM reads API keys from environment variables: 
# Required for respective providers
export OPENAI_API_KEY=sk-...
export ANTHROPIC_API_KEY=sk-ant-...
export GEMINI_API_KEY=...
Never commit API keys to version control. Always use environment variables or secure secret management systems.

​
Using with qckfx

Once LiteLLM is running, configure qckfx to use it:

​
Environment Variable Method

export LLM_BASE_URL=http://localhost:8001
qckfx -m claude-sonnet-4 "Analyze this codebase"

​
Command Line Method

qckfx --url http://localhost:8001 -m gemini-2.5-pro "Generate documentation"

​
Agent Configuration Method

{
  "environment": "local",
  "defaultModel": "claude-sonnet-4"
}
Then run with the base URL: 
export LLM_BASE_URL=http://localhost:8001
qckfx -a my-agent.json "Execute with LiteLLM"

​
Production Deployment

For production use, consider these additional configurations:

​
Docker Compose

Create docker-compose.yml: 
version: '3.8'
services:
  litellm:
    build: .
    ports:
      - "8001:8001"
    environment:
      - OPENAI_API_KEY=${OPENAI_API_KEY}
      - ANTHROPIC_API_KEY=${ANTHROPIC_API_KEY}
      - GEMINI_API_KEY=${GEMINI_API_KEY}
    restart: unless-stopped
    volumes:
      - ./config.yaml:/app/config.yaml:ro
Run with: 
docker-compose up -d

​
Kubernetes Deployment

Create litellm-deployment.yaml: 
apiVersion: apps/v1
kind: Deployment
metadata:
  name: litellm
spec:
  replicas: 2
  selector:
    matchLabels:
      app: litellm
  template:
    metadata:
      labels:
        app: litellm
    spec:
      containers:
      - name: litellm
        image: your-registry/litellm:latest
        ports:
        - containerPort: 8001
        env:
        - name: OPENAI_API_KEY
          valueFrom:
            secretKeyRef:
              name: api-keys
              key: openai
        # ... other API keys
---
apiVersion: v1
kind: Service
metadata:
  name: litellm-service
spec:
  selector:
    app: litellm
  ports:
  - port: 8001
    targetPort: 8001
  type: LoadBalancer

​
Authentication and Security

​
API Key Management

For enhanced security, LiteLLM supports API key authentication: 
# Add to config.yaml
general_settings:
  master_key: your-master-key
  
# Use with qckfx
export LLM_API_KEY=your-master-key
qckfx -m claude-sonnet-4 "Secure request"

​
Rate Limiting

Configure rate limits in config.yaml: 
general_settings:
  rpm_limit: 100  # Requests per minute
  tpm_limit: 10000  # Tokens per minute

​
Monitoring and Logging

​
Request Logging

Enable detailed logging: 
general_settings:
  set_verbose: true
  json_logs: true

​
Health Checks

LiteLLM provides health check endpoints: 
# Check if LiteLLM is running
curl http://localhost:8001/health

# Get model information
curl http://localhost:8001/v1/models

​
Troubleshooting

​
Connection Issues

$ qckfx --url http://localhost:8001 -m claude-sonnet-4 "test"
Error: connect ECONNREFUSED 127.0.0.1:8001
Solutions:
  • Verify LiteLLM is running: docker ps
  • Check port mapping: docker port <container-id>
  • Ensure firewall allows port 8001

​
Model Not Found

Error: Model 'unknown-model' not found
Solutions:
  • Check available models: curl http://localhost:8001/v1/models
  • Verify model name in config.yaml
  • Restart LiteLLM after configuration changes

​
API Key Issues

Error: Invalid API key for provider
Solutions:
  • Verify environment variables are set
  • Check API key format for each provider
  • Ensure keys have necessary permissions

​
Configuration Errors

Error: Failed to load configuration
Solutions:
  • Validate YAML syntax in config.yaml
  • Check file permissions
  • Review LiteLLM logs: docker logs <container-id>

​
Alternative Providers

While LiteLLM is recommended, you can also use:

​
OpenRouter

export LLM_BASE_URL=https://openrouter.ai/api/v1
export LLM_API_KEY=sk-or-...
qckfx -m anthropic/claude-3.5-sonnet "Direct OpenRouter usage"

​
Direct Provider APIs

# Direct Anthropic
export LLM_BASE_URL=https://api.anthropic.com
export LLM_API_KEY=sk-ant-...

# Direct OpenAI
export LLM_BASE_URL=https://api.openai.com/v1
export LLM_API_KEY=sk-...
⌘I