Using Legion with local LLMs

View Source

Legion talks to models through ReqLLM, so any provider ReqLLM supports, Legion can run on - including servers on your own machine. Ollama has a native :ollama provider and is the quickest start; vLLM has one too, and anything that speaks the OpenAI Chat Completions API works through the :openai provider with a custom base_url - ReqLLM's Model Specs guide shows how, and the model support page lists every provider it ships with.

Ollama

  1. Install Ollama and pull a model. The server listens on localhost:11434:
ollama pull qwen3:8b
  1. Point Legion at it, globally or per agent:
# config/config.exs
config :legion, :config, %{model: "ollama:qwen3:8b"}

# or per agent, which wins over the global setting
def config, do: %{model: "ollama:qwen3:8b"}

Local model names are not in ReqLLM's catalog, so the first request logs an "unverified model" warning. It only means there is no pricing metadata; silence it with config :req_llm, warn_unverified_models: false.

  1. Verify in iex -S mix, same as in Installation:
Legion.execute(PingAgent, "Return the sum of 2 and 2")
#=> {:ok, "4"}

The first call is slow while Ollama loads the model; later ones are not.

A remote Ollama host is provider config, not model config:

config :req_llm, :ollama, base_url: "http://gpu-box:11434/v1"

Picking a model

Everything above the model stays the same - tools, sandboxes, store, rate limiter, legion_web. What changes is how much the model has to carry: it must return structured JSON on every step, write working Lua (or Elixir) against tool source it has just read, and take in a long system prompt that includes every tool's source. So pick a model that handles structured output and code well, with a context window to match, and make sure the server is actually configured to use that window; local servers tend to default to a small one and truncate silently. A lower max_iterations keeps a struggling model from looping for long.