Why Self-Host Your LLM Models on a VPS
Self-hosting a model server means keeping your prompts and sensitive data off commercial APIs, removing per-token billing, and setting the model, its version, and its quantization yourself. On a VPS, you expose a private API to your internal applications (chatbots, RAG, code assistants) without any leak to the outside. Ollama stands out for its radical simplicity: one command to download and run a model, a clean API, automatic memory management. LocalAI positions itself as a 'drop-in' replacement for the OpenAI API: it exposes the same endpoints (chat, embeddings, images, audio) and accepts multiple backends and model formats. The choice comes down to a minimalist experience versus the broadest possible compatibility.
The Benefits of a Self-Hosted LLM
- Prompts and confidential data that never leave your VPS
- No per-token billing, predictable cost tied only to the server
- A private API wired directly into your internal apps
- Free choice of the model, its size, and its quantization level
- Compatibility with existing SDKs via an OpenAI-style API
- Ideal for RAG: pair the LLM with your database and a self-hosted search engine
Prerequisites: Quantized Models and Realistic RAM
Without a GPU, stick to lightweight quantized models. A 3B model in GGUF Q4 runs on a VPS with 4 vCPUs and 8 GB of RAM with acceptable latency for testing; a 7B/8B Q4 model needs 8 to 16 GB of RAM and a good CPU to stay usable. Beyond that, on pure CPU, latency becomes prohibitive: for real-time on large models, a VPS with a GPU is essential. Above all, plan for a generous disk: the weights of several models quickly add up to tens of GB. You'll need Docker and Compose, a persistent volume for the models, a subdomain if you expose the API, and a reverse proxy with authentication.
Deploy Ollama (or LocalAI) on a VPS
Provision Storage and the Models Volume
Create a dedicated volume (/srv/ollama/models) on a sufficiently large disk. Since the weights are large and reusable, they must persist outside the container to avoid re-downloading on every restart.
Launch the Container
Start ollama/ollama (or localai/localai), mounting the models volume and binding port 11434/8080 locally. On a VPS without a GPU, CPU mode is automatic; with a GPU, enable the appropriate runtime.
Download a Model
With Ollama, run docker compose exec ollama ollama pull llama3.2:3b. With LocalAI, declare the model in the gallery or drop the GGUF file into the models folder, then verify its loading in the logs.
Test the API
Make a local call: curl http://127.0.0.1:11434/api/generate for Ollama, or the OpenAI-compatible endpoint POST /v1/chat/completions for LocalAI. Confirm that generation works before any exposure.
Expose Behind an Authenticated Reverse Proxy
Proxy llm.yourdomain.com to the local port with Caddy or Nginx for TLS, and add an authentication layer (an API key in a header or basic auth). An LLM API open to the Internet is a costly entry point that should never be left uncontrolled.
Connect Your Applications
Point your existing SDKs to your private base_url. Since LocalAI exposes the OpenAI API, most libraries work by simply changing the URL and the key; on the Ollama side, use its native API or its compatible endpoint.
| Criterion | Ollama | LocalAI |
|---|---|---|
| Philosophy | Simplicity, one command per model | Drop-in replacement for the OpenAI API |
| OpenAI API compatibility | Compatible endpoint available | Native and very complete |
| Supported modalities | Text, embeddings, vision (depending on model) | Text, embeddings, images, audio, TTS |
| Model management | `ollama pull`, very smooth | Gallery + files, more manual |
| Backends / formats | Mainly GGUF | Multiple backends and formats |
| Getting started | Very fast | More configuration |
| GPU / CPU support | Both, easy switching | Both, broad hardware support |
| Ideal use case | Start fast, prototype | Migrate an OpenAI app to self-hosting |
On a VPS without a GPU, the secret to smoothness is quantization: a model in Q4_K_M offers the best size/quality trade-off and fits in RAM where the non-quantized version collapses. Also limit the context window to the strict minimum (for example 4096 tokens): an oversized context multiplies memory consumption and latency with no real benefit for most tasks.
