Do I need a GPU to run LocalAI?

No. LocalAI is built on llama.cpp, which runs efficiently on CPU using SIMD instructions (AVX2/AVX-512 on Intel, NEON on ARM). A VPS with 2–4 GB RAM and 2–4 vCPU is enough for 3B–7B models. GPU acceleration is optional and supported if your server has a CUDA-compatible GPU.

Which models can I run?

LocalAI supports any model in GGUF format, which covers virtually all open-source LLMs: Llama 3.x, Mistral 0.7–7B, Phi-3/3.5/4 Mini, Qwen 2.5, DeepSeek-R1, Gemma 2, Falcon, and more. Pull a model with one API call — LocalAI downloads it from Hugging Face automatically.

How does LocalAI compare to Ollama?

Both run LLMs locally on CPU. LocalAI's advantage is its broader compatibility: it supports more model backends (not just llama.cpp), image generation, speech-to-text (Whisper), and embeddings from a single API. Ollama is simpler to set up and has a larger default model library. Both expose an OpenAI-compatible API.

Yes. All inference happens on your VPS — no request is ever sent to an external API. The models are stored in a Docker volume on your own disk. LocalAI has no telemetry by default (MIT licence, fully auditable source code).

What are the RAM requirements?

A 3B model at Q4 quantisation needs roughly 2 GB RAM. A 7B model needs ~4 GB. A 13B model needs ~8 GB. For the smallest ServOrbit VPS plans (2 GB), the Phi-3 Mini (3.8B at Q4) runs well and delivers strong reasoning performance relative to its size.

Build, host and run AI solutions.

LocalAI

Run open-source LLMs on CPU — no GPU required. OpenAI-compatible API, 200+ models, self-hosted in one command.

2 GB (4 GB recommended for 7B+ models) RAM 2 vCPU (4 vCPU recommended) Port 8080 Available

Source code

Tech stack

DockerGollama.cpp

Minimum RAM2 GB (4 GB recommended for 7B+ models)

Minimum CPU2 vCPU (4 vCPU recommended)

Default port8080

Compatible OSChoice of Linux distributions

LocalAI (MIT, ~47k GitHub stars) is a free, open-source alternative to the OpenAI API. It runs on consumer-grade hardware without a GPU — any VPS with 2 GB RAM can serve Llama 3, Mistral, Phi-3, Qwen and 200+ other models via an OpenAI-compatible REST API. Your existing code that calls `openai.ChatCompletion.create()` works without modification: change the base URL to your VPS and all API calls are routed locally, with zero per-token cost and complete data privacy.

Deployed on a ServOrbit VPS, LocalAI becomes your private LLM backend: an endpoint your team's applications, RAG pipelines and agent frameworks call instead of OpenAI — with no cloud dependency, no usage limits, and no data leaving your infrastructure. LocalAI supports text generation, function calling, image generation (Stable Diffusion), speech-to-text (Whisper), and text embeddings — all from a single container.

Key features

OpenAI-compatible REST API — drop-in replacement for OpenAI, works with every LangChain/LlamaIndex/LiteLLM SDK unchanged.

CPU-only inference — runs on any VPS without a GPU using llama.cpp; optimised SIMD paths for Intel and ARM.

200+ models supported — Llama 3, Mistral, Phi-3, Qwen 2.5, DeepSeek, Gemma 2, Falcon, and any GGUF-format model.

Multi-modal: text generation, function/tool calling, embeddings, image generation (Stable Diffusion), and speech-to-text (Whisper).

Pull models by name — `curl http://localhost:8080/models/apply -d '{"id":"llama-3.2-3b-instruct:q4_0"}'` downloads and activates any supported model.

MIT licence — fully auditable, no telemetry, air-gap capable; v4.6.2 (released 2026-07-06) adds distributed cluster mode and prefix-cache-aware routing.

Composable with LiteLLM and Open WebUI — LocalAI serves the inference layer; LiteLLM adds multi-provider routing; Open WebUI adds the chat UI.

When to use this solution?

1

Private LLM backend for your applications

Replace your OpenAI API calls with a LocalAI endpoint on your VPS. Your SaaS, internal tool, or RAG pipeline sends requests to `http://your-vps:8080/v1/chat/completions` — identical JSON, identical SDK, zero cost per token. Ideal for high-volume use cases where per-token API pricing becomes a bottleneck.

2

Air-gapped inference for sensitive data

For legal, medical or financial applications where data cannot leave your infrastructure, LocalAI runs entirely offline once the model is downloaded. No requests ever reach an external API. The GGUF model files are stored in a Docker volume on your own disk.

3

Embeddings and semantic search without an API key

Run embedding models (nomic-embed-text, mxbai-embed-large) locally alongside Qdrant on the same VPS to build a complete semantic search or RAG pipeline. No OpenAI embedding API cost, no data leaving your server, no rate limits.

Deploy LocalAI on your VPS

Guide optimized for ServOrbit Cloud VPS.

01

Create the VPS

Order a VPS on ServOrbit with at least 2 GB RAM and Ubuntu 22.04 or 24.04. For comfortable multi-user usage or larger models (7B+), 4 GB RAM is recommended. CPU count matters for inference speed: 4 vCPU reduces generation latency noticeably on 3–7B models.

02

Deploy LocalAI

Connect via SSH and run: docker run -d --restart=always -p 8080:8080 -v local-ai:/build/models --name localai localai/localai:latest. The image (about 900 MB) starts an OpenAI-compatible API server. The first request for a model triggers an automatic download to the persistent volume.

03

Pull a model

Download your first model with a single API call: curl http://localhost:8080/models/apply -H 'Content-Type: application/json' -d '{"id":"llama-3.2-3b-instruct:q4_0"}'. LocalAI fetches the GGUF file and registers it. The 3B Q4 quantisation fits in 2 GB RAM; the 7B Q4 needs ~4 GB.

04

Make your first API call

Test the API: curl http://localhost:8080/v1/chat/completions -H 'Content-Type: application/json' -d '{"model":"llama-3.2-3b-instruct:q4_0","messages":[{"role":"user","content":"Hello!"}]}'. The response is identical to OpenAI's — change the base URL in your existing code and everything works.

05

Point your apps at LocalAI

In any OpenAI SDK, set base_url='http://your-vps-ip:8080/v1' and api_key='not-needed'. In LangChain: ChatOpenAI(base_url=..., api_key='x'). In LiteLLM: add a localai/ prefix. In Open WebUI: configure an OpenAI connection with your VPS URL. Your existing code needs no other change.

06

Optional: protect the endpoint

LocalAI is open by default. To add authentication, set LOCALAI_API_KEY=your-secret environment variable when running the container. For HTTPS, place Caddy in front: ai.yourdomain.com { reverse_proxy localhost:8080 } — Caddy issues the Let's Encrypt cert automatically. Restrict port 8080 to your office IP using ufw if needed.

Frequently asked questions

LocalAI is a free, open-source (MIT) server that runs large language models locally on any CPU. It exposes an OpenAI-compatible REST API, so your existing code works without modification — just change the API base URL to your server.

Complementary solutions

Artificial Intelligence

Activate LocalAI on your infrastructure.

Dedicated Cloud VPS — IPv4 included, European datacenter, support included. Your data never leaves your server.

Recommended configuration: 2 GB (4 GB recommended for 7B+ models) RAM · 2 vCPU (4 vCPU recommended)