Do I need a GPU to run LocalAI?

No. LocalAI is built on llama.cpp, which runs efficiently on CPU using SIMD instructions (AVX2/AVX-512 on Intel, NEON on ARM). A VPS with 2–4 GB RAM and 2–4 vCPU is enough for 3B–7B models. GPU acceleration is optional and supported if your server has a CUDA-compatible GPU.

Which models can I run?

LocalAI supports any model in GGUF format, which covers virtually all open-source LLMs: Llama 3.x, Mistral 0.7–7B, Phi-3/3.5/4 Mini, Qwen 2.5, DeepSeek-R1, Gemma 2, Falcon, and more. Pull a model with one API call — LocalAI downloads it from Hugging Face automatically.

How does LocalAI compare to Ollama?

Both run LLMs locally on CPU. LocalAI's advantage is its broader compatibility: it supports more model backends (not just llama.cpp), image generation, speech-to-text (Whisper), and embeddings from a single API. Ollama is simpler to set up and has a larger default model library. Both expose an OpenAI-compatible API.

Yes. All inference happens on your VPS — no request is ever sent to an external API. The models are stored in a Docker volume on your own disk. LocalAI has no telemetry by default (MIT licence, fully auditable source code).

What are the RAM requirements?

A 3B model at Q4 quantisation needs roughly 2 GB RAM. A 7B model needs ~4 GB. A 13B model needs ~8 GB. For the smallest ServOrbit VPS plans (2 GB), the Phi-3 Mini (3.8B at Q4) runs well and delivers strong reasoning performance relative to its size.

Build, host and run AI solutions.

LocalAI

Run open-source LLMs on CPU — no GPU required. OpenAI-compatible API, 200+ models, self-hosted in one command.

2 GB (4 GB recommended for 7B+ models) RAM 2 vCPU (4 vCPU recommended) Port 8080 Available

Source code

Tech stack

DockerGollama.cpp

Minimum RAM2 GB (4 GB recommended for 7B+ models)

Minimum CPU2 vCPU (4 vCPU recommended)

Default port8080

Compatible OSubuntu-24.04

LocalAI (MIT, ~47k GitHub stars) is a free, open-source alternative to the OpenAI API. It runs on consumer-grade hardware without a GPU — any VPS with 2 GB RAM can serve Llama 3, Mistral, Phi-3, Qwen and 200+ other models via an OpenAI-compatible REST API. Your existing code that calls `openai.ChatCompletion.create()` works without modification: change the base URL to your VPS and all API calls are routed locally, with zero per-token cost and complete data privacy.

Deployed on a ServOrbit VPS, LocalAI becomes your private LLM backend: an endpoint your team's applications, RAG pipelines and agent frameworks call instead of OpenAI — with no cloud dependency, no usage limits, and no data leaving your infrastructure. LocalAI supports text generation, function calling, image generation (Stable Diffusion), speech-to-text (Whisper), and text embeddings — all from a single container.

Key features

OpenAI-compatible REST API — drop-in replacement for OpenAI, works with every LangChain/LlamaIndex/LiteLLM SDK unchanged.

CPU-only inference — runs on any VPS without a GPU using llama.cpp; optimised SIMD paths for Intel and ARM.

200+ models supported — Llama 3, Mistral, Phi-3, Qwen 2.5, DeepSeek, Gemma 2, Falcon, and any GGUF-format model.

Multi-modal: text generation, function/tool calling, embeddings, image generation (Stable Diffusion), and speech-to-text (Whisper).

Pull models by name — curl http://localhost:8080/models/apply -d '{"id":"llama-3.2-3b-instruct:q4_0"}' downloads and activates any supported model.

MIT licence — fully auditable, no telemetry, air-gap capable; v4.6.2 (released 2026-07-06) adds distributed cluster mode and prefix-cache-aware routing.

Composable with LiteLLM and Open WebUI — LocalAI serves the inference layer; LiteLLM adds multi-provider routing; Open WebUI adds the chat UI.

When to use this solution?

1

Private LLM backend for your applications

Replace your OpenAI API calls with a LocalAI endpoint on your VPS. Your SaaS, internal tool, or RAG pipeline sends identical JSON with an identical SDK, at zero cost per token. The endpoint is not public — LocalAI listens on the loopback only, because it ships with no authentication whatsoever: code running on the VPS itself calls http://127.0.0.1:8080/v1/chat/completions, and reaching it from another machine takes an SSH tunnel. Ideal for high-volume use cases where per-token API pricing becomes a bottleneck.

2

Air-gapped inference for sensitive data

For legal, medical or financial applications where data cannot leave your infrastructure, LocalAI runs entirely offline once the model is downloaded. No requests ever reach an external API. The GGUF model files are stored in a Docker volume on your own disk.

3

Embeddings and semantic search without an API key

Run embedding models (nomic-embed-text, mxbai-embed-large) locally alongside Qdrant on the same VPS to build a complete semantic search or RAG pipeline. No OpenAI embedding API cost, no data leaving your server, no rate limits.

Deploy LocalAI on your VPS

Guide optimized for ServOrbit Cloud VPS.

01

Create the VPS

Order a ServOrbit VPS with at least 2 GB of RAM; the system installed is Ubuntu 24.04. For comfortable multi-user use, or for models from 7 billion parameters upwards, 4 GB is recommended. Core count matters for speed: 4 vCPU noticeably reduce generation latency on models of 3 to 7 billion parameters.

02

Deploy LocalAI

Installing from your client area creates the localai/localai:latest container — an image of about 900 MB — mounts the local-ai volume on /build/models, and publishes the OpenAI-compatible API server on the loopback interface, at 127.0.0.1:8080. No model ships with it: the first request naming a model triggers its download into the persistent volume.

03

Connect for the first time

LocalAI opens without any credentials at all: your first move is to go through the model gallery and download one, since the installation provides none. The OpenAI-compatible API answers at the same address and is protected by no key — as long as the service listens only on the loopback interface, only users with SSH access to the VPS can reach it.

04

Pull a model

A single API call is enough: curl http://127.0.0.1:8080/models/apply -H 'Content-Type: application/json' -d '{"id":"llama-3.2-3b-instruct:q4_0"}'. LocalAI fetches the GGUF file and registers the model. The Q4 quantisation of a 3 billion parameter model fits in 2 GB of RAM; a 7 billion one calls for about 4.

05

Make your first API call

Test the API from the VPS: curl http://127.0.0.1:8080/v1/chat/completions -H 'Content-Type: application/json' -d '{"model":"llama-3.2-3b-instruct:q4_0","messages":[{"role":"user","content":"Hello!"}]}'. The response has exactly the shape of OpenAI's: changing the base URL in your existing code is all it takes to make it work.

06

Point your apps at LocalAI

From the VPS itself, set base_url='http://127.0.0.1:8080/v1' and api_key='not-needed' in any OpenAI SDK — that is the normal case, since the application consuming LocalAI runs on the same machine. From your own workstation, open an SSH tunnel — ssh -L 8080:127.0.0.1:8080 root@<your-vps-ip> — and point at http://localhost:8080/v1. The port may be reassigned at install time: use the one shown on the app's card in your client area, 8080 being only the catalogue value. In LangChain that gives ChatOpenAI(base_url=..., api_key='x'); in LiteLLM, prefix the model with localai/; in Open WebUI, declare an OpenAI connection with that same base URL. Your existing code needs no other change.

07

Protect the endpoint

LocalAI is open by default: no key is required. The application knows how to demand one through the LOCALAI_API_KEY environment variable, but the catalogue installation does not set one — the endpoint is therefore unauthenticated, which is precisely why it is never published behind a domain: an inference engine open to the internet is your CPU time and your bill at the disposal of whoever finds it. To expose a public API, publish an authenticating gateway instead — LiteLLM for instance — and keep LocalAI behind it, on the loopback.

Frequently asked questions

LocalAI is a free, open-source (MIT) server that runs large language models locally on any CPU. It exposes an OpenAI-compatible REST API, so your existing code works without modification — just change the API base URL to your server.

Embed the deploy button

Maintaining a project that uses LocalAI? This button lets your readers deploy it on a VPS in one click, without reading Docker documentation.

Markdown

[![Deploy LocalAI on ServOrbit](https://servorbit.com/brand/deploy/button.svg)](https://servorbit.com/vps-cloud?template=local-ai&utm_source=deploy-badge&utm_medium=referral&utm_campaign=local-ai)

HTML

<a href="https://servorbit.com/vps-cloud?template=local-ai&utm_source=deploy-badge&utm_medium=referral&utm_campaign=local-ai"><img src="https://servorbit.com/brand/deploy/button.svg" alt="Deploy LocalAI on ServOrbit" height="40"></a>

The button points to a VPS order with the template preselected. The image is served from servorbit.com — nothing to host on your side.

Complementary solutions

Artificial Intelligence

Activate LocalAI on your infrastructure.

Dedicated Cloud VPS — IPv4 included, European datacenter, support included. Your data never leaves your server.

Recommended configuration: 2 GB (4 GB recommended for 7B+ models) RAM · 2 vCPU (4 vCPU recommended)