How to Host PrivateGPT on a VPS

Why self-host PrivateGPT on a VPS

Concrete benefits of self-hosting

• Full confidentiality: documents, embeddings and the Qdrant index never leave the VPS
• Sourced answers: every reply relies on passages retrieved from your files, not on the model's general memory
• No API costs when the model runs locally through Ollama or llama.cpp
• Ingestion of PDF, DOCX, PPTX, EPUB, CSV, Markdown, .mbox mail archives and even images or audio, organized into collections
• A Messages-style API (modeled on the Anthropic API specification) to plug PrivateGPT into your internal tools
• Easier compliance for sensitive data (GDPR, professional secrecy): no additional AI processor

PrivateGPT and Ollama: two complementary roles

PrivateGPT vs Ollama: who does what

Scroll the table

Hardware and software requirements

Pre-installation checklist

• A VPS with at least 8 GB of RAM (16 GB for a 9-billion-parameter model) and 4 vCPUs
• Up-to-date Debian 12 or Ubuntu 24.04, SSH access with sudo
• Docker installed (docker version responds) or, failing that, uv with Python 3.11
• 30 GB of free disk, NVMe preferred, for models and local_data/
• A subdomain (for example docs.your-domain.com) pointing to the VPS IP
• An active firewall: only SSH, 80 and 443 open, ports 8080 and 11434 stay closed

Deploying PrivateGPT in local mode

1. 01

   Install Ollama and pull the models

   Install Ollama with the official script, then download a CPU-friendly LLM and the recommended embedding model:

   curl -fsSL https://ollama.com/install.sh | sh
   ollama pull qwen3.5:4b
   ollama pull mxbai-embed-large

   By default Ollama only listens on 127.0.0.1:11434: keep it that way, only PrivateGPT needs to reach it. This is the only step that needs Internet access; once the models are in place, inference runs offline.

2. 02

   Run PrivateGPT with Docker

   The official zylonai/private-gpt image ships the core package. On Linux the container must share the host network to reach Ollama on localhost, and a volume must keep the data, otherwise ingested documents vanish on restart:

   mkdir -p /opt/privategpt/local_data
   docker run -d --name privategpt --restart unless-stopped \
     --network host \
     -e OPENAI_API_BASE=http://localhost:11434/v1 \
     -v /opt/privategpt/local_data:/home/worker/app/local_data \
     zylonai/private-gpt:latest

   At startup, PrivateGPT calls GET /v1/models on Ollama and registers the models it finds automatically: no configuration file is required. The OPENAI_EMBEDDING_API_BASE variable is only needed if your embeddings are served elsewhere.

3. 03

   Ingest your documents by collection

   Send your files to the ingestion API, grouping them into collections, for example one collection per client file or per department:

   curl -X POST http://localhost:8080/v1/artifacts/ingest \
     -F "file=@/srv/documents/contrat-cession.pdf" \
     -F "collection=legal"
   curl "http://localhost:8080/v1/artifacts/list?collection=legal"

   PrivateGPT splits, vectorizes and indexes each file in Qdrant. Formats without a dedicated reader are read as plain text.

4. 04

   Query the corpus with citations

   Test first in the Workbench (http://localhost:8080/ui), then plug your tools into the Messages API. Document search goes through the built-in semantic_search_v1 tool, restricted to one collection:

   curl http://localhost:8080/v1/messages \
     -H "Content-Type: application/json" \
     -d '{"model": "qwen3.5:4b",
          "messages": [{"role": "user", "content": "What are the conditions precedent in the contract?"}],
          "tools": [{"name": "search_docs", "type": "semantic_search_v1",
                     "context": [{"type": "ingested_artifact", "context_filter": {"collection": "legal"}}]}]}'

5. 05

   Set up the reverse proxy, SSL and authentication

   The Workbench has no access control: the documentation reserves it for local use. Expose it only behind Nginx or Caddy, with a Let's Encrypt certificate and basic authentication:

   server {
       listen 443 ssl;
       server_name docs.your-domain.com;
       ssl_certificate     /etc/letsencrypt/live/docs.your-domain.com/fullchain.pem;
       ssl_certificate_key /etc/letsencrypt/live/docs.your-domain.com/privkey.pem;
       auth_basic           "Restricted access";
       auth_basic_user_file /etc/nginx/.htpasswd;
       location / { proxy_pass http://127.0.0.1:8080; }
   }

   If only two or three people use it, an SSH tunnel (ssh -L 8080:127.0.0.1:8080 user@your-vps) avoids any public exposure at all.

6. 06

   Lock down access

   PrivateGPT listens on 0.0.0.0 by default, so host networking publishes port 8080 on every interface. Close it at the firewall so that only the reverse proxy is reachable:

   ufw default deny incoming
   ufw allow OpenSSH
   ufw allow 80,443/tcp
   ufw enable

   Your users then go through https://docs.your-domain.com and query the corpus in natural language.

Back up and verify that no data leaves

ss -tnp state established

Troubleshooting: the most common errors