AnythingLLM on VPS: Full Guide with Pitfalls to Avoid

Why self-host AnythingLLM on a VPS

AnythingLLM is first and foremost a complete RAG engine: document ingestion, chunking, vectorization, semantic search and generation, all within a browser-accessible multi-user interface. The heart of the matter is the data: financial reports, contracts, product documentation, support knowledge base. These files have no business transiting through a third-party SaaS ingestion provider.

On a VPS, the document embedder, the vector database (LanceDB built in by default) and conversation history stay in your Docker volumes — never duplicated elsewhere without your consent. You control who accesses which workspace, freely choose the LLM (cloud API or local model via Ollama), and avoid the ingestion quotas of SaaS offerings that cap the number of pages or files.

For an agency managing several clients, each workspace becomes a watertight silo: partitioned documents, role-based permissions, no leakage between teams. For a company, compliance is the priority: data stays on your infrastructure, in your jurisdiction, under your backup policy.

The concrete benefits of a self-hosted AnythingLLM

Confidential documents indexed locally, never sent to a third-party ingestion service.
Workspaces partitioned by client or team, with fine-grained role management (Admin, Manager, Default).
Vector database of your choice: embedded LanceDB to start, external Chroma or Qdrant for large corpora.
Connection to 20+ LLM providers, including a local Ollama for zero-cloud and zero inference cost.
No limit on the number of documents, pages or workspaces ingested.
Simple and portable backup: the entire state fits in a single storage volume to archive or replicate.
Built-in no-code AI agents: create automation pipelines without leaving the interface.
Full REST API to integrate AnythingLLM into your own applications or automation scripts.

Hardware and software requirements

The AnythingLLM container is reasonable on resources, but embedding a large corpus consumes CPU and RAM significantly. Aim for 2 vCPU / 2 GB of RAM to start with a few hundred documents and a cloud LLM API. Scale up to 4 vCPU / 8 GB if you index thousands of documents, use a local embedding model, or run Ollama on the same host.

Plan for 15 to 20 GB of disk as a minimum: vectors and the document cache grow quickly, especially with dense PDFs or multilingual corpora. Add margin if you also deploy Qdrant or Ollama on the same VPS.

On the software side, you need:
- Docker and Docker Compose (v2 recommended)
- A domain name or subdomain pointing to your VPS (e.g. chat.your-domain.com)
- Port 443 open inbound on your firewall
- An LLM API key (OpenAI, Anthropic, Mistral…) if you are not using local Ollama

Important note on versions: before any docker pull command, read the Version Pinning section below. Pulling :latest without precaution can permanently destroy your embeddings.

Version pinning: critical and irreversible

Never use the :latest tag in production. A docker pull mintplexlabs/anythingllm:latest can overwrite the internal encryption key of embeddings stored in your volume, making all your vectorized documents permanently unreadable — with no recovery possible.

This behavior is documented as a breaking change since March 2026 (GitHub issue #5256).

The rule to follow: always pin a fixed version in your docker-compose.yml:

image: mintplexlabs/anythingllm:v1.8.4

Before updating, read the release notes for each intermediate version, back up your full storage volume, and test the version upgrade on a copy before applying it to production.

Deploy AnythingLLM with Docker and HTTPS

Create the directory structure and permissions

Over SSH on your VPS, create the working directory and data volume:

mkdir -p /opt/anythingllm/storage
cd /opt/anythingllm
chmod -R 777 storage

The container runs with a dedicated UID (non-root): 777 permissions on storage are necessary for the internal process to write the vector database and document uploads.

Write docker-compose.yml with a pinned version

Create /opt/anythingllm/docker-compose.yml. Note the fixed version tag — do not use :latest:

services:
  anythingllm:
    image: mintplexlabs/anythingllm:v1.8.4
    container_name: anythingllm
    restart: unless-stopped
    ports:
      - "3001:3001"
    volumes:
      - ./storage:/app/server/storage
    env_file:
      - .env
    cap_add:
      - SYS_ADMIN

Then create .env in the same folder with at minimum:

JWT_SECRET=change-me-to-a-long-random-string
STORAGE_DIR=/app/server/storage
LLM_PROVIDER=openai
OPEN_AI_KEY=sk-your-openai-key

Replace LLM_PROVIDER and the key according to your provider. For local Ollama, see the next step.

Configure Ollama if you opt for 100% local

If Ollama is running on the same host as the AnythingLLM container, do not use localhost — from inside the container, localhost refers to the container itself, not the host.

Correct URL by OS:
- Linux: http://172.17.0.1:11434 (default Docker bridge address)
- macOS / Windows: http://host.docker.internal:11434

In your .env:

LLM_PROVIDER=ollama
OLLAMA_BASE_PATH=http://172.17.0.1:11434
OLLAMA_MODEL_PREF=llama3.2
EMBEDDING_ENGINE=ollama
EMBEDDING_BASE_PATH=http://172.17.0.1:11434
EMBEDDING_MODEL_PREF=nomic-embed-text

Verify that Ollama is listening on 0.0.0.0 (not just 127.0.0.1) by checking OLLAMA_HOST=0.0.0.0 in its systemd unit or environment variable.

Start the container and create the admin account

Launch the service:

docker compose up -d
docker compose logs -f

Wait for the lines [server] Listening on port 3001 and [database] Migration complete. Then access http://your-ip:3001 from your workstation (temporary access, to be closed after setting up the reverse proxy). The configuration wizard guides you to create the administrator account and choose the embedding model and LLM.

Set up the reverse proxy with HTTPS

With Nginx or Caddy, expose AnythingLLM behind your domain.

Caddy example (simplest, automatic Let's Encrypt):

chat.your-domain.com {
    reverse_proxy localhost:3001
    request_body {
        max_size 100MB
    }
}

Nginx example (server block):

server {
    listen 443 ssl;
    server_name chat.your-domain.com;
    ssl_certificate /etc/letsencrypt/live/chat.your-domain.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/chat.your-domain.com/privkey.pem;
    client_max_body_size 100M;
    location / {
        proxy_pass http://127.0.0.1:3001;
        proxy_set_header Host $host;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
    }
}

The client_max_body_size directive (Nginx) or max_size (Caddy) is essential to allow uploading large PDFs or document archives.

Create a workspace, ingest documents and test

In the interface, click '+ New workspace', give it a name, then drag a few PDFs into the upload area. Launch the embedding (button 'Save and embed'). Once done, ask a question in the chat — answers should cite sources extracted from your files.

If citations do not appear, check in the workspace settings that RAG mode is enabled (option 'Chat mode' → 'Query'), and that the number of chunks returned is greater than 0.

Enable multi-user mode and secure access

In Settings → Multi-User Mode, enable multi-user mode. Invite your collaborators by email and assign roles: Admin (full access), Manager (workspace management), Default (use only).

Restrict each user to only the workspaces that concern them. From the admin interface, you can also configure SSO or external authentication if your LLM provider supports it.

Troubleshooting: the three most common pitfalls

These three issues come up regularly in the AnythingLLM community. Knowing them before deploying will save you hours of debugging.

Pitfall 1 — :latest destroys your embeddings (critical, irreversible)

If you update your container with docker pull mintplexlabs/anythingllm:latest, a new version may overwrite the encryption key of embeddings stored in your volume. Result: all your vectorized documents become unreadable, with no recovery possible. This breaking change has been documented since March 2026 (issue #5256). Solution: always pin a fixed version tag (v1.8.4, v1.9.x…) in your docker-compose.yml and never pull without a prior backup of the storage volume.

Pitfall 2 — Gemini agents are broken since v1.16

Since AnythingLLM v1.16, the Gemini provider produces a connection error during streaming that stays open indefinitely, blocking the agent (issue #6153, still open). Workaround: disable the Gemini provider in LLM settings and use instead an OpenAI-compatible endpoint or switch to Ollama. Do not try to debug at the network level — the issue is in AnythingLLM's stream handling.

Pitfall 3 — Ollama unreachable from the container

If Ollama is installed directly on the host (outside Docker), configuring http://localhost:11434 in AnythingLLM does not work: from inside the container, localhost points to the container itself, not the host. Use http://172.17.0.1:11434 on Linux (Docker bridge IP) or http://host.docker.internal:11434 on macOS/Windows. Also check that Ollama listens on 0.0.0.0 and not only 127.0.0.1 (OLLAMA_HOST=0.0.0.0 variable in the systemd service).

For large corpora (more than 10,000 chunks), do not stick with embedded LanceDB: deploy Qdrant in a neighboring container on the same Docker network and point AnythingLLM to it via the VECTOR_DB=qdrant and QDRANT_ENDPOINT=http://qdrant:6333 variables. Qdrant handles millions of vectors better, offers metadata filtering and remains independently queryable, which makes debugging your semantic searches easier. The Qdrant volume is also easy to back up: a simple docker cp or volume snapshot is enough.

AnythingLLM vs Open WebUI: which AI workspace is right for you?

	AnythingLLM	Open WebUI
Primary focus	Document RAG + AI agents	Chat UI for LLMs
RAG / document ingestion	Built-in (PDF, Word, URL, Notion, GitHub…)	Basic file upload
AI agent builder	Yes (no-code)	No
LLM providers	20+ (Ollama, OpenAI, Anthropic, Mistral…)	Ollama + OpenAI-compatible
Multi-user roles	Admin / Manager / Default	Basic user management
REST API	Full document & chat API	Limited
Vector store	LanceDB built-in + swappable (Qdrant, Chroma…)	External via RAG config
RAM (API-only, no local LLM)	~512 MB	~256 MB
:latest risk	Critical — destroys embeddings	Less documented
Licence	MIT	MIT