LLM Providers¶

Configure which AI model powers Lumen.

Quick start¶

Set your API key and launch:

export OPENAI_API_KEY="sk-..."
lumen-ai serve penguins.csv

Lumen auto-detects the provider from environment variables.

Specify provider explicitly¶

To use a specific provider, pass the --provider flag:

lumen-ai serve penguins.csv --provider anthropic
lumen-ai serve penguins.csv --provider google --model 'gemini-3.8-flash'
lumen-ai serve penguins.csv --provider kilo --model 'openai/gpt-4o-mini'
lumen-ai serve penguins.csv --provider ollama --model 'qwen3.6:35b-a3b'
lumen-ai serve penguins.csv --provider mlx

For more CLI options, see the CLI guide.

Different models per agent¶

Use cheap models for simple tasks, powerful models for complex tasks:

import lumen.ai as lmai

model_config = {
    "default": {"model": "gpt-6-luna"},  # Cheap for most agents
    "sql": {"model": "gpt-4.1"},           # Powerful for SQL
    "vega_lite": {"model": "gpt-4.1"},     # Powerful for charts
    "deck_gl": {"model": "gpt-4.1"},       # Powerful for 3D maps
    "analyst": {"model": "gpt-4.1"},       # Powerful for analysis
}

llm = lmai.llm.OpenAI(model_kwargs=model_config)
ui = lmai.ExplorerUI(data='penguins.csv', llm=llm)
ui.servable()

Agent names map to model types: SQLAgent → "sql", VegaLiteAgent → "vega_lite", etc.

Track token usage and cost¶

LLM instances collect provider-reported token usage in llm.usage. Configure usage_pricing with USD rates per million tokens to estimate cost for each model; prices are not bundled with Lumen and must be kept current for your provider and account.

import asyncio

from lumen.ai.llm import OpenAI


async def main():
    llm = OpenAI(
        model_kwargs={"default": {"model": "gpt-4o-mini"}},
        usage_pricing={
            "gpt-4o-mini": {"input": 0.15, "cached": 0.075, "output": 0.60},
        },
    )
    with llm.capture_usage() as request_usage:
        await llm.invoke([{"role": "user", "content": "Say hello."}])

    print(request_usage.input_tokens, request_usage.output_tokens)
    print(request_usage.cached_tokens, request_usage.cost_usd)
    print(llm.usage.cost_usd)  # Total across calls made with this LLM instance


asyncio.run(main())

Set OPENAI_API_KEY before running this example. The usage_pricing keys must match the model names returned by the provider, including any provider prefix. Each model needs input and output rates; cached defaults to the input rate when omitted. For Anthropic, cache_write can price cache creation separately and also defaults to the input rate. Cached reads and writes are included in input_tokens; cached_tokens counts reads only. Inspect request_usage.records or llm.usage.records for individual responses and their model, token counts, and cost.

llm.usage aggregates the lifetime of that LLM instance. llm.capture_usage() creates a scoped collector for calls within the context, including calls from child async tasks; use separate contexts to attribute concurrent requests. A scope includes all provider calls made within it, such as tool rounds and structured-output retries. Consume a streaming response fully to receive its final usage report. If a provider does not return usage, no record is added. If a model has no pricing entry, its tokens are recorded but cost_usd is None for that record and for any collector containing it.

Configure temperature¶

Lower temperature = more deterministic. Higher = more creative.

model_config = {
    "sql": {
        "model": "gpt-4.1",
        "temperature": 0.1,  # Deterministic SQL
    },
    "chat": {
        "model": "gpt-5.4-mini",
        "temperature": 0.4,  # Natural conversation
    },
}

Recommended ranges: 0.1 (SQL) to 0.4 (chat).

Disabling temperature¶

Some newer models (e.g. reasoning models) reject a temperature argument. Set temperature=None to omit it from requests entirely, falling back to the provider's own default:

llm = lmai.llm.Anthropic(temperature=None)

This works on every provider. Local backends (MLX, Llama.cpp) fall back to their built-in sampler default when None.

Supported providers¶

For installation and API key setup instructions, see the Installation guide.

Cloud providers¶

Local providers¶

Recommended local models:

• General purpose: qwen3.6:35b-a3b, qwen3.8:27b, gemma4:26b, gpt-oss:20b
• Smaller machines: qwen3.5:9b
• Coding: qwen3-coder:30b
• Reasoning: nemotron-3.5-lightning:30b

Codex CLI and Claude Code¶

Use a locally authenticated Codex CLI or Claude Code CLI as Lumen's LLM provider. This lets an individual user run Lumen with an existing CLI subscription, without supplying an API key to Lumen.

Before starting Lumen, install the relevant CLI, make sure its executable is on your PATH, and sign in to it once:

codex login

claude auth login

Start Lumen without loading any data:

lumen-ai serve --provider codex-cli

lumen-ai serve --provider claude-code

The CLI's configured default model is used unless you select one explicitly. For example, Claude Code provides a stable sonnet alias:

lumen-ai serve --provider claude-code --model sonnet

The same --model option accepts model identifiers available to the authenticated Codex CLI account.

To start with a dataset, add its path before the provider arguments:

lumen-ai serve penguins.csv --provider codex-cli
lumen-ai serve penguins.csv --provider claude-code

GitHub Copilot CLI¶

Use a locally authenticated GitHub Copilot CLI as Lumen's LLM provider. It works with any Copilot plan, including Copilot Free.

Install the CLI, make sure copilot is on your PATH, and sign in once:

copilot login

Start Lumen with it, with or without a dataset:

lumen-ai serve --provider copilot-cli
lumen-ai serve penguins.csv --provider copilot-cli

Copilot picks the model by default. Lumen then keeps one Copilot process running and sends every request to it, which avoids several seconds of CLI startup per request. If your plan lets you choose a model, pass a name listed by copilot help config with --model; Lumen then starts the CLI once per request, which is slower. Plans limited to automatic model selection, such as Copilot Free, only accept --model auto; any other name fails with Model ... from --model flag is not available.

Antigravity CLI¶

Use a locally authenticated Google Antigravity CLI (agy) as Lumen's LLM provider. Requests count against your Antigravity account's usage limits.

Install the CLI, then run agy once and sign in with your Google account:

curl -fsSL https://antigravity.google/cli/install.sh | bash
agy

Start Lumen with it, with or without a dataset:

lumen-ai serve --provider antigravity-cli
lumen-ai serve penguins.csv --provider antigravity-cli

Lumen uses gemini-3.8-flash-low by default, a low-cost Flash model and the fastest one in testing: a penguins query and chart took about 40 seconds, compared with about 90 seconds using Antigravity's own default. Run agy models to list the other models you can pass with --model.

Router / Gateway providers¶

AWS Bedrock options:

• AnthropicBedrock - Optimized for Claude models using Anthropic's SDK
• Bedrock - Universal access to all Bedrock models (Claude, Llama, Mistral, Titan, etc.)

Kilo¶

Kilo provides an OpenAI-compatible API for routing requests to models from multiple providers.

import lumen.ai as lmai

llm = lmai.llm.Kilo(
    model_kwargs={
        "default": {"model": "kilo-auto/free"},
        "sql": {"model": "anthropic/claude-sonnet-4.6"},
        "vega_lite": {"model": "openai/gpt-5.4-mini"},
    }
)

ui = lmai.ExplorerUI(data='penguins.csv', llm=llm)
ui.servable()

OpenRouter¶

OpenRouter provides an OpenAI-compatible API for routing requests to models from multiple providers.

import lumen.ai as lmai

llm = lmai.llm.OpenRouter(
    model_kwargs={
        "default": {"model": "openai/gpt-6-luna"},
        "sql": {"model": "deepseek/deepseek-v4-pro-0813"},
        "vega_lite": {"model": "qwen/qwen3.8-flash"},
    }
)

ui = lmai.ExplorerUI(data='penguins.csv', llm=llm)
ui.servable()

Advanced configuration¶

API keys¶

Each provider has three ways to supply an API key, resolved in this order:

1. api_key param — pass it directly at instantiation
2. api_key_env_var — a class variable naming the environment variable to read from; defaults to the value from PROVIDER_ENV_VARS for that provider
3. PROVIDER_ENV_VARS — the module-level dict that is the source of truth for env var names; changing it updates both api_key_env_var defaults and get_available_llm() auto-detection

llm = lmai.llm.OpenAI(api_key="sk-...")

import os
os.environ["OPENAI_API_KEY"] = "sk-..."
llm = lmai.llm.OpenAI()  # api_key is auto-populated from OPENAI_API_KEY

If your key is stored under a different environment variable name, there are three ways to override it:

lmai.llm.OpenAI.api_key_env_var = "MY_OPENAI_KEY"
llm = lmai.llm.OpenAI()  # now reads from MY_OPENAI_KEY

class MyOpenAI(lmai.llm.OpenAI):
    api_key_env_var = "MY_OPENAI_KEY"

llm = MyOpenAI(model_kwargs={"default": {"model": "gpt-5.4-mini"}})

import lumen.ai.llm as llm_module

llm_module.PROVIDER_ENV_VARS["openai"] = "MY_OPENAI_KEY"

Option 3 also affects get_available_llm(), which uses PROVIDER_ENV_VARS to auto-detect which provider to use at startup.

The default values for each provider are:

Checking available models¶

models() is a classmethod you can call directly to inspect what models are available from a provider without instantiating the LLM:

print(lmai.llm.OpenAI.models())
# {'gpt-5.4-mini', 'gpt-4.1', 'gpt-4.1-nano', ...}

print(lmai.llm.Anthropic.models())
# {'claude-haiku-4-5', 'claude-sonnet-4-6', 'claude-opus-4-6', ...}

print(lmai.llm.OpenRouter.models())
# {'openai/gpt-6-luna', 'qwen/qwen3.8-flash', 'z-ai/glm-5.3-flash', ...}

Providers that don't support listing models (e.g. AzureOpenAI, LiteLLM) return an empty set.

If you implement a custom provider subclass, override models() to support it:

class MyProvider(lmai.llm.Llm):
    api_key_env_var = "MY_PROVIDER_API_KEY"

    @classmethod
    def models(cls) -> set[str]:
        import requests
        api_key = os.environ.get(cls.api_key_env_var)
        if not api_key:
            return set()
        resp = requests.get("https://my-provider.com/models",
                            headers={"Authorization": f"Bearer {api_key}"})
        return {m["id"] for m in resp.json()["data"]}

Custom endpoints¶

Override default API endpoints:

llm = lmai.llm.OpenAI(
    api_key='...',
    endpoint='https://your-custom-endpoint.com/v1'
)

Managed Identity (Azure)¶

Use Azure Active Directory authentication:

from azure.identity import DefaultAzureCredential, get_bearer_token_provider

token_provider = get_bearer_token_provider(
    DefaultAzureCredential(),
    "https://cognitiveservices.azure.com/.default"
)

llm = lmai.llm.AzureOpenAI(
    api_version="2024-02-15-preview",
    endpoint="https://your-resource.openai.azure.com/",
    model_kwargs={
        "default": {
            "model": "gpt4o-mini",
            "azure_ad_token_provider": token_provider
        }
    }
)

Prompt caching (Anthropic)¶

Anthropic caches the static prefix of each request (tools → system → messages) so repeated context isn't reprocessed, lowering latency and input-token cost. It is controlled by the cache parameter:

llm = lmai.llm.Anthropic(cache="5m")  # default; also "1h" or None to disable

• "5m" (default) — 5-minute cache; refreshes for free on each hit.
• "1h" — 1-hour cache; costs more on write, useful for slower-cadence sessions.
• None — disabled.

Fallback models (LiteLLM)¶

Automatically retry with backup models if primary fails:

llm = lmai.llm.LiteLLM(
    model_kwargs={
        "default": {"model": "gpt-5.4-mini"}
    },
    fallback_models=[
        "gpt-5.4-nano",
        "anthropic/claude-haiku-4-5",
        "gemini/gemini-3.8-flash"
    ]
)

Remote Ollama server¶

Connect to Ollama running on another machine:

llm = lmai.llm.Ollama(
    endpoint='http://your-server:11434/v1',
    model_kwargs={"default": {"model": "qwen3.6:35b-a3b"}}
)

AI Navigator (Local)¶

Connect to Anaconda AI Navigator running on your machine:

llm = lmai.llm.AINavigator()

By default, it uses http://localhost:8080/v1.

MLX (Apple Silicon)¶

Run models natively on Apple Silicon Macs using the MLX framework. Requires pip install mlx-lm.

Two modes of operation:

mlx_lm.server \
  --model mlx-community/Qwen3.5-9B-MLX-4bit \
  --port 8080 \
  --max-tokens 8192 \
  --chat-template-args '{"enable_thinking":false}'

llm = lmai.llm.MLX(
    endpoint="http://localhost:8080/v1",
    model_kwargs={"default": {"model": "mlx-community/Qwen3.5-9B-MLX-4bit"}},
)

llm = lmai.llm.MLX(
    endpoint=None,  # disables server mode
    model_kwargs={"default": {"model": "mlx-community/Qwen3.5-9B-MLX-4bit"}},
)

Key parameters:

Recommended models for Apple Silicon:

AI Catalyst (Enterprise)¶

Connect to Anaconda AI Catalyst running on your company's infrastructure:

# Requires AI_CATALYST_BASE_URL and AI_CATALYST_API_KEY env vars
llm = lmai.llm.AICatalyst()

Or configure explicitly:

llm = lmai.llm.AICatalyst(
    endpoint='https://your-company.anacondaconnect.com/api/v1/model-servers/your-server/v1',
    api_key='...'
)

Model types¶

Agent class names convert to model types automatically:

Conversion rule: remove "Agent" suffix, convert to snake_case.

Additional model types:

• edit - Used when fixing errors
• ui - Used for UI initialization

Model string formats¶

Different providers use different model string formats:

• OpenAI: "gpt-6-luna", "gpt-5.6-luna", "gpt-6-sol", "gpt-5.4-nano"
• Anthropic: "claude-haiku-4-5", "claude-sonnet-4-6", "claude-opus-4-6"
• OpenRouter: "openai/gpt-6-luna", "qwen/qwen3.8-flash", "z-ai/glm-5.3-flash", "deepseek/deepseek-v4.1-flash"
• Google: "gemini-3.8-flash", "gemini-3.1-flash-lite"
• Mistral: "mistral-large-latest", "mistral-small-latest"
• Azure: "your-deployment-name" (use your Azure deployment name)
• Bedrock: "us.anthropic.claude-haiku-4-5-20251001-v1:0", "us.anthropic.claude-sonnet-4-6", "us.amazon.nova-2-lite-v1:0"
• LiteLLM: "gpt-5.4-mini" (OpenAI), "anthropic/claude-sonnet-4-6" (Anthropic), "gemini/gemini-3.8-flash" (Google)

For OpenRouter and LiteLLM, use the provider/model format for routed models.

Troubleshooting¶

"API key not found" - Set environment variable or pass api_key= in Python.

Wrong model used - Model type names must be snake_case: "sql" not "SQLAgent".

High costs - Use gpt-5.4-mini or claude-haiku-4-5 for default, reserve gpt-5.4 or claude-sonnet-4-6 for critical tasks (sql, vega_lite, analyst).

Slow responses - Local models are slower than cloud APIs. Use cloud providers when speed matters.

AWS credentials not found - For Bedrock, ensure AWS credentials are configured. See Installation guide.

Best practices¶

Use efficient models elsewhere:

• default - Simple tasks work well with gpt-5.4-mini or claude-haiku-4-5
• chat - Conversation works with smaller models

Avoid reasoning models in dialog interfaces:

• Reasoning models (o1, o1-mini, gemini-3.0-pro) are significantly slower
• They're designed for single, complex queries, not interactive chat
• For Lumen's dialog interface, use standard models (mistral-small-latest, gpt-5.4-mini, claude-sonnet-4-6)
• Reserve reasoning models for batch processing or one-off complex analyses

Set temperature by task:

• 0.1 for SQL (deterministic)
• 0.3-0.4 for analysis and chat
• 0.5-0.7 for creative tasks

Test before deploying:

• Different models behave differently. Test with real queries.