How to Host LocalAI on a VPS

Where Ollama focuses on text, LocalAI aims for broad compatibility with the OpenAI API: it exposes /v1/chat/completions, /v1/embeddings, /v1/images/generations and even audio transcription, on a single API. For a team that already has code written against the OpenAI SDK, LocalAI is a near-transparent replacement: you change the base URL and the key, the rest works. On a VPS, you get this versatility without external dependency or per-call cost. This is especially relevant when you need text generation, vectors for RAG and, occasionally, images all at once, without multiplying providers and keys.

Since LocalAI is versatile, its needs depend on the functions enabled. The figures below are measured minimums — a more powerful VPS reduces response times but does not change feasibility.

Text only, CPU, 7B model (e.g. Mistral-7B-GGUF): 8 GB of RAM, 4 vCPU, 30 GB of disk. At this size, a /v1/chat/completions call takes 5 to 15 seconds depending on the quantization (Q4 is the standard trade-off). A 3B model fits in 4 GB of RAM.

Text + images (7B + Stable Diffusion): 16 GB of RAM, GPU recommended, 50 GB of disk. Without a GPU, generating one image takes several minutes on a CPU — prefer a GPU VPS for regular use and the SINGLE_ACTIVE_BACKEND=true option to avoid saturation when mixing chat and images on the same CPU host.

Required software: Docker 24+ and Docker Compose v2 (docker compose without a hyphen), a dedicated subdomain (e.g. ia.your-domain.com), port 443 open, a DNS record pointing to the VPS IP.

LocalAI supports several formats, but the GGUF format (produced by llama.cpp) is the recommended standard for CPU: it is quantized (reduces RAM usage), portable and well supported by the gallery.

GGUF vs other formats: GGUF formats are the only ones directly loadable via llama.cpp without prior conversion. Diffusers models (images) and ONNX (lightweight embeddings) have their own backends and are not interchangeable with GGUFs.

Gallery vs manual download: the gallery is the simplest option — it downloads the GGUF and automatically generates the configuration YAML. For a model not listed in the gallery (a fine-tuned model, a custom GGUF), drop the file into ./models/ and create the YAML manually:

name: my-model
backend: llama
parameters:
  model: my-model.Q4_K_M.gguf
context_size: 4096

List loaded models:

curl https://ia.your-domain.com/v1/models \
  -H 'Authorization: Bearer your-secret-key'

Models whose YAML is present in ./config/ appear in the list, even if their GGUF has not yet been downloaded — in that case, the first call triggers the download.

Since LocalAI is compatible with the OpenAI API, migrating an existing application comes down to two changes: the base URL and the key.

Python (openai SDK):

import openai

client = openai.OpenAI(
    base_url="https://ia.your-domain.com/v1",
    api_key="your-secret-key"
)

response = client.chat.completions.create(
    model="mistral-7b-instruct-v0.2.Q4_K_M",
    messages=[{"role": "user", "content": "Summarize this text: ..."}]
)
print(response.choices[0].message.content)

Node.js (openai SDK):

import OpenAI from 'openai';

const client = new OpenAI({
  baseURL: 'https://ia.your-domain.com/v1',
  apiKey: 'your-secret-key'
});

const completion = await client.chat.completions.create({
  model: 'mistral-7b-instruct-v0.2.Q4_K_M',
  messages: [{ role: 'user', content: 'Hello' }]
});
console.log(completion.choices[0].message.content);

Direct curl:

curl https://ia.your-domain.com/v1/embeddings \
  -H 'Authorization: Bearer your-secret-key' \
  -H 'Content-Type: application/json' \
  -d '{"model": "all-minilm", "input": "Text to vectorize"}'

LangChain, LlamaIndex and most RAG frameworks accept an openai_api_base or base_url parameter — the change is identical.

404 model not found or model not loaded: the model name passed in model: does not match any YAML manifest in ./config/. Check with GET /v1/models: the displayed name must be used exactly as shown. If the model does not appear, verify the YAML is in the mounted folder and restart the container.

Very slow inference or killed (OOM): the model exceeds available RAM. A 7B Q4 model requires approximately 6 GB of free RAM at load time. Check with docker stats localai during an inference. Solutions: switch to a lighter quantization (Q3_K_S or Q2_K), reduce the context_size in the YAML, or upgrade the VPS.

SINGLE_ACTIVE_BACKEND error: LocalAI is configured with SINGLE_ACTIVE_BACKEND=true and a request attempts to load a second model while another is active. This is the expected behavior on CPU. Wait for the first inference to finish, or disable the option if your VPS has enough RAM for multiple simultaneous models.

GPU not detected (CUDA device not found): verify you are using the latest-gpu-nvidia-cuda-12 image, that the NVIDIA driver is installed on the host (nvidia-smi), and that the container has GPU access (gpus: all in docker-compose.yaml under the deploy: key). Without all three conditions, LocalAI silently falls back to CPU.

401 Unauthorized: the API_KEY variable is set in docker-compose.yaml but the Authorization: Bearer <key> header is missing or incorrect in the call. All endpoints, including /v1/models, require the header once API_KEY is set.