Integrating OpenClaw with Open-Source LLMs: Llama 2, Mistral, and More

If you’re running OpenClaw and looking to reduce your API costs or gain more control over your model choices, integrating with open-source LLMs like Llama 2 or Mistral is a powerful next step. The typical setup for OpenClaw involves connecting to commercial APIs like Anthropic’s Claude or OpenAI’s GPT models. While convenient, these can become expensive, especially for high-volume or experimental use cases. The good news is that OpenClaw’s architecture is flexible enough to accommodate locally hosted or self-managed LLMs, provided you set up an OpenAI-compatible API endpoint.

Affiliate Disclosure: As an Amazon Associate, we earn from qualifying purchases. This means we may earn a small commission when you click our links and make a purchase on Amazon. This comes at no extra cost to you and helps support our site.

The Problem with Direct Integration

OpenClaw doesn’t natively support direct interaction with model weights or common open-source inference servers like `text-generation-inference` or `ollama` out of the box. Its core design assumes an OpenAI-like API interface for model communication. This means you can’t just point OpenClaw to a local Llama 2 model file and expect it to work. You need an intermediary layer that translates OpenClaw’s OpenAI-compatible requests into something your local LLM can understand, and then translates the LLM’s responses back into an OpenAI-compatible format.

Setting Up Your OpenAI-Compatible Endpoint

The most robust and widely supported solution for creating an OpenAI-compatible endpoint for open-source LLMs is to use a project like vLLM or text-generation-webui (specifically its API mode). For production-like environments or high throughput, `vLLM` is often preferred due to its superior inference performance, especially with larger batch sizes. For simpler setups or if you’re already familiar with `text-generation-webui`, its API is perfectly adequate.

Let’s assume you’re using `vLLM` for its efficiency. First, ensure you have a machine with a powerful GPU (NVIDIA preferred) and sufficient VRAM for your chosen model. A Llama 2 7B model requires at least 8-10GB of VRAM, while a 70B model needs 80GB or more, often necessitating multiple GPUs. Install `vLLM`:

pip install vllm

Then, you can start an API server for a model, for example, Mistral-7B-Instruct-v0.2:

python -m vllm.entrypoints.api_server --model mistralai/Mistral-7B-Instruct-v0.2 --port 8000 --host 0.0.0.0

This command downloads the specified model (if not already cached) and exposes an OpenAI-compatible API endpoint on `http://0.0.0.0:8000`. You can then test it with `curl`:

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "mistralai/Mistral-7B-Instruct-v0.2",
    "messages": [
      {"role": "user", "content": "Hello, how are you?"}
    ],
    "max_tokens": 50
  }'

The `model` name in the `vLLM` API call is crucial. It directly corresponds to the model identifier you passed when starting `vLLM` (e.g., `mistralai/Mistral-7B-Instruct-v0.2`). OpenClaw will use this value.

Configuring OpenClaw to Use Your Local LLM

Once your OpenAI-compatible endpoint is running, you need to tell OpenClaw to use it instead of its default commercial API. This is done by modifying your OpenClaw configuration. You’ll need to create or edit the `~/.openclaw/config.json` file. If it doesn’t exist, create it. If it does, be careful not to overwrite existing settings.

Add an `openai` section to your configuration that points to your local `vLLM` endpoint:

{
  "general": {
    "log_level": "INFO"
  },
  "openai": {
    "api_key": "sk-not-required",
    "base_url": "http://localhost:8000/v1",
    "model_map": {
      "default": "mistralai/Mistral-7B-Instruct-v0.2",
      "fast": "mistralai/Mistral-7B-Instruct-v0.2",
      "code": "codellama/CodeLlama-7b-Instruct-hf"
    }
  },
  "anthropic": {
    "api_key": "YOUR_CLAUDE_API_KEY"
  }
}

Let’s break down these critical fields:

• api_key: Even though `vLLM` typically doesn’t require an API key, OpenClaw’s OpenAI client expects one. A placeholder like `”sk-not-required”` or any non-empty string will suffice.
• base_url: This is the most important part. It must point to the root of your `vLLM`’s OpenAI-compatible API, specifically ending with `/v1`. If your `vLLM` server is on a different machine, replace `localhost` with its IP address or hostname.
• model_map: This defines the logical model names OpenClaw uses (e.g., `default`, `fast`, `code`) and maps them to the actual model identifiers that your `vLLM` server expects. In our example, `mistralai/Mistral-7B-Instruct-v0.2` is the model `vLLM` is serving. If you run multiple `vLLM` instances for different models (e.g., one for Mistral, one for CodeLlama), you would map them here. This is where you gain flexibility; you could point “code” to a local CodeLlama instance, “fast” to a smaller, faster model, and “default” to your general-purpose choice.

It’s vital to understand that OpenClaw will now prioritize the `openai` section if its `base_url` is set. If you leave the `anthropic` or other provider sections in your `config.json`, they will still be available, but your default OpenClaw commands will now use the locally hosted model mapped under the `openai` provider.

Non-Obvious Insight: Model Mapping and Prompts

While OpenClaw will now technically talk to your local LLM, not all open-source models are instruction-tuned in the same way as commercial ones like Claude or GPT. Many open-source models require specific chat templates or prompt formats (e.g., Llama 2 uses `[INST] … [/INST]` tags, Mistral has its own format). OpenClaw’s prompt engineering is generally designed for commercial models. When using open-source models, especially instruction-tuned ones, you might find that your OpenClaw prompts need to be slightly adjusted or that the model’s responses are less coherent than expected. The `vLLM` server (and other similar API wrappers) typically handle the conversion of OpenAI’s chat message format into the model’s native instruction format, but this isn’t always perfect.

Experimentation is key here. If you’re seeing poor results, consider simplifying your prompts or looking at the specific prompt format recommended by the open-source model’s creators. Sometimes, a simpler, more direct prompt works better with a less sophisticated instruction-following model.

Another point: while `claude-haiku-4-5` might be cheap and good for many tasks on Anthropic’s platform, the performance characteristics of local open-source models are different. A 7B parameter open-source model running on a consumer GPU might be slower than a commercial API call, but its cost is zero beyond hardware and electricity. For tasks that require high throughput and can tolerate slightly lower quality, a local 7B or 13B model can be incredibly cost-effective.

Limitations

This approach hinges on having dedicated hardware. You need a machine with a powerful GPU and sufficient VRAM. Running a 7B parameter model on a Raspberry Pi is simply not feasible for anything close to real-time inference. Even a VPS without a dedicated GPU will struggle immensely, falling back to CPU inference which is orders of magnitude slower. This setup is best suited for a dedicated server, a powerful workstation, or a cloud instance with GPU acceleration. For 7B models, 16GB of system RAM and 8GB+ of VRAM are a good baseline. For larger models, these requirements scale significantly.

Frequently Asked Questions