PrivateGPT on a VPS: analyze confidential documents locally

Why your professional documents cannot go through the cloud

A lawyer pasting a merger contract into ChatGPT, an accountant submitting a tax return to Claude, an HR manager summarising a disciplinary file via the OpenAI API: in all three cases, data flows to third-party servers. GDPR article 28 is clear: the data controller remains responsible even when a sub-processor — here OpenAI or Anthropic — receives the data.

AI Act article 50, in force since 2 August 2026, adds another layer: general-purpose generative AI systems must be declared or fall under an explicit exemption. Using a US provider's API to process personal data without prior declaration now carries concrete regulatory risk.

The immediate legal path is on-premise. A root VPS running PrivateGPT answers questions about your documents without ever calling api.openai.com or api.anthropic.com. You can verify this in real time with tcpdump — this is an architectural property, not a contractual promise.

What you gain with on-premise PrivateGPT

Zero outbound data: the model runs locally via llama-cpp-python, no network calls to an AI provider — verifiable by firewall or tcpdump.
GDPR article 28 compliance: no additional sub-processor, no DPA to negotiate, no transfer outside the EU.
AI Act article 50 exemption: a self-hosted LLM with no public access does not fall under the obligation to declare general-purpose generative AI systems.
Professional secrecy preserved: contracts, tax filings, medical records and HR files stay within your legal perimeter.
RAG on your own documents: PrivateGPT indexes your PDF, DOCX, TXT and CSV files and answers while citing source passages — not a generic chatbot.
No recurring subscription: the cost is the VPS; the Mistral-7B-Instruct model is open-source.
CPU-only viable: no GPU required — a VPS with 8 GB of RAM and an NVMe SSD is enough for Mistral-7B Q4_K_M.

PrivateGPT vs Ollama: two tools that do different things

Ollama is a generic LLM server: it downloads models, exposes an OpenAI-compatible API and answers prompts. It is an excellent tool, but it does not read your files — it answers from its conversation context, not your documents.

PrivateGPT (repo github.com/zylon-ai/private-gpt) is a RAG engine (Retrieval-Augmented Generation): it indexes your documents into a local vector store (ChromaDB), then for each question retrieves the relevant passages and injects them into the LLM's context. The answer cites its sources. This mechanism is what makes the tool useful on an 80-page contract or a 12-annex tax file: without RAG, the model hallucinates; with RAG, it reads.

PrivateGPT vs Ollama — key differences

Criterion	PrivateGPT	Ollama
Primary use case	Q&A on your own documents (RAG)	Generic LLM serving (API)
File indexing	PDF, DOCX, TXT, CSV native	No — manual context
Web interface	Yes (port 8080)	No (API only)
Vector store	Local ChromaDB	None
Source citation	Yes — passage + filename	No
GPU required	No — CPU-only viable	No — CPU-only viable

Prerequisites: choosing the right VPS before you start

Memory requirements are imposed by the model, not by PrivateGPT itself. The official documentation and field measurements converge on two thresholds:

- Mistral-7B-Instruct Q4_K_M (PrivateGPT's default model): 8 GB of RAM minimum. Below this, the process is killed by the kernel (OOM) when loading the model.
- 13B model (e.g. Llama-2-13B Q4_K_M): 16 GB of RAM minimum. Response quality is noticeably better on long and technical texts.

NVMe SSD recommended: indexing a 50-page PDF generates vector embeddings — on an NVMe disk this takes a few seconds; on an HDD or shared SATA SSD it can block for several minutes.

Pre-installation checklist

VPS with 8 GB of RAM minimum (16 GB if targeting a 13B model).
Debian 12 or Ubuntu 22.04/24.04 — PrivateGPT's official Docker images target these distributions.
Docker and Docker Compose installed (docker compose version must return v2.x).
NVMe SSD for the data volume (fast PDF indexing).
SSH root or sudo access to the VPS.
Port 8080 not publicly exposed at this stage — PrivateGPT listens locally by default.

Deploy PrivateGPT with Docker Compose

Clone the repository and prepare the structure

git clone https://github.com/zylon-ai/private-gpt.git
cd private-gpt
mkdir -p local_data/private_gpt docs

The docs/ folder will receive the documents you want to index. The local_data/ folder contains the ChromaDB vector store and downloaded models — never mount it on an unencrypted network volume.

Configure the environment

cp .env.example .env

In .env, the values to check:

DOCKER_COMPOSE_PROFILE=ollama
PRIVATEGPT_SERVER_HOST=127.0.0.1
PRIVATEGPT_SERVER_PORT=8080

The ollama profile starts PrivateGPT in CPU mode with the llama-cpp-python backend.

Start the service and download the model

docker compose --profile ollama up -d
docker compose logs -f privategpt

When you see Application startup complete, the service is ready. The first request may take 15–30 seconds (model loading into RAM).

Ingest your documents

cp /path/to/merger-contract.pdf docs/
curl -X POST http://127.0.0.1:8080/v1/ingest/file \
  -H 'Content-Type: multipart/form-data' \
  -F 'file=@docs/merger-contract.pdf'

Supported formats: PDF, DOCX, TXT and CSV. The MAX_INPUT_SIZE variable (.env) caps the size per file.

Access the interface via an SSH tunnel

PrivateGPT listens on 127.0.0.1:8080, never exposed directly to the internet. To access it from your workstation:

ssh -L 8080:127.0.0.1:8080 [email protected]

Then open http://localhost:8080 in your browser.

Query a contract

In the web interface, make sure Query documents mode is selected (not LLM Chat, which ignores the vector store). Example:

> "What are the conditions precedent set out in article 4?"

PrivateGPT returns the answer with cited source passages. Via the API:

curl -X POST http://127.0.0.1:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"messages":[{"role":"user","content":"What are the conditions precedent in article 4?"}],"use_context":true}'

Post-installation: secure and back up

Team access with Nginx + basic authentication

# /etc/nginx/sites-available/privategpt
server {
    listen 443 ssl;
    server_name privategpt.your-domain.com;
    ssl_certificate     /etc/letsencrypt/live/privategpt.your-domain.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/privategpt.your-domain.com/privkey.pem;
    auth_basic "Restricted access";
    auth_basic_user_file /etc/nginx/.htpasswd;
    location / { proxy_pass http://127.0.0.1:8080; }
}

Backing up critical data — two folders constitute the complete state: local_data/ (ChromaDB vector store + models) and docs/ (your source documents).

Verifying no outbound traffic to AI APIs:

tcpdump -i any host api.openai.com or host api.anthropic.com

No packets should appear.

Model too slow? Switch to a smaller quantisation. If response time exceeds 90 seconds on Mistral-7B, download the Q3_K_S version (~3.0 GB, ~20% faster on CPU) and update the LLM_HF_MODEL_FILE variable in .env. Quality drops slightly on very long texts, but remains sufficient for precise questions on a well-structured contract. Conversely, if you have 16 GB of RAM available, mistral-7b-instruct-v0.2.Q8_0.gguf (~7.7 GB) delivers more nuanced answers on ambiguous clauses.

Most common errors

OOM at launch — docker ps shows Exited (137). Verify RAM with free -h. Switch to Q3_K_S model if insufficient.

Ingestion stalled — check permissions: chown -R 1000:1000 local_data/. The container runs as non-root (UID 1000).

Response time over 60 seconds — the model is paging to swap. Check with htop during a request. Increase VPS RAM or use a smaller model.

Results without source citation — verify Query documents mode is selected (not LLM Chat), and check GET http://127.0.0.1:8080/v1/ingest/list.

Container restarting in a loop after an update — delete local_data/chroma_db/ and re-index your documents.

PrivateGPT in production: an infrastructure decision, not just a Docker run

Installing PrivateGPT on a VPS takes less than an hour. Maintaining it in operational conditions — model updates, backups, team access management, RAM consumption monitoring — requires the same reflexes as any business service. That is precisely what a root VPS provides: the freedom to process your data under your own rules, with a responsibility perimeter you control end to end.