Why LiteLLM is the missing layer for self-hosted AI stacks
Ollama is great for running local models. Open WebUI wraps it in a chat interface. But the moment you start building applications — not just chatting — you need something in between: a proxy that normalises the API surface, tracks costs per project, and lets you swap models without touching app code. That is exactly what LiteLLM does. It sits between your application and any LLM backend, exposing a single OpenAI-compatible endpoint. Your code stays the same whether you route to Llama 3.1 on Ollama, Claude 3.5 Haiku, or GPT-4o.
What LiteLLM gives you out of the box
- Unified OpenAI-compatible API — works with every SDK, library and tool that targets the OpenAI API.
- 100+ provider integrations: Ollama, Anthropic, OpenAI, Azure OpenAI, Mistral, Cohere, Bedrock and more.
- Virtual API keys with per-key spend limits and rate limits — issue keys to teams and cap monthly costs.
- Built-in dashboard at /ui with real-time spend analytics, per-model breakdowns and request logs.
- Load balancing and automatic failover across multiple provider endpoints or model replicas.
- Apache 2.0 licence — no telemetry, no phoning home, fully air-gap capable when paired with Ollama.
Requirements
At least 1 GB of RAM and Ubuntu 22.04. Docker and the Compose plugin installed. Port 4000 open, or a Caddy/Nginx reverse proxy for HTTPS. That is everything — LiteLLM itself is a Python proxy and needs around 256 MB; PostgreSQL takes most of the remaining RAM budget. For teams with concurrent traffic, 2 GB is more comfortable.
Deploy LiteLLM on a VPS
Provision the VPS
Order a VPS Solo (2 vCPU, 4 GB, Ubuntu 22.04): LiteLLM needs only 1 GB, and the smallest plan in the catalogue serves twice that. SSH in and install Docker:
curl -fsSL https://get.docker.com | sh. The Compose plugin is included.Generate secrets
Run
openssl rand -hex 32to generate LITELLM_MASTER_KEY — this unlocks the dashboard and all admin API endpoints. Create/opt/litellm/.envwith:LITELLM_MASTER_KEY,UI_USERNAME,UI_PASSWORD,DB_NAME=litellm,DB_USER=litellm,DB_PASSWORD=<strong-password>.Write docker-compose.yml
Create
/opt/litellm/docker-compose.ymlwith two services:db(postgres:16-alpine, env vars from .env, volume litellm_db) andlitellm(image ghcr.io/berriai/litellm pinned to an explicit version and its@sha256:digest, nevermain-latest— see "Supply chain" below —, port 4000:4000, depends_on db, DATABASE_URL + LITELLM_MASTER_KEY + UI_USERNAME + UI_PASSWORD, volume litellm_config:/app/data). Rundocker compose up -d.Open the dashboard
Navigate to
http://<vps-ip>:4000/uiand log in with your UI_USERNAME / UI_PASSWORD. You will land on the LiteLLM dashboard. Go to Models to add your first provider. For a local Ollama instance on the same VPS, set model name toollama/llama3.1and API base tohttp://host.docker.internal:11434. For Anthropic, paste your API key and addanthropic/claude-3-5-haiku-20241022.Test the gateway
From your workstation, test:
curl http://<vps-ip>:4000/v1/chat/completions -H "Authorization: Bearer <LITELLM_MASTER_KEY>" -H 'Content-Type: application/json' -d '{"model": "ollama/llama3.1", "messages": [{"role": "user", "content": "Hello"}]}'. You should get a streamed response back — routed through LiteLLM to your Ollama.Issue team API keys
In the dashboard under API Keys, click New Key. Give it a description (e.g. 'frontend-team'), set
max_budget: 50(USD/month) and an optionaltpm_limit. Share this key with the team — they point their OpenAI SDK athttp://<vps-ip>:4000and use this key. Spend is tracked per key in the dashboard.Add HTTPS with Caddy
Install Caddy:
apt install -y caddy. Edit/etc/caddy/Caddyfile:llm.yourdomain.com { reverse_proxy localhost:4000 }. Reload:systemctl reload caddy. Close port 4000 in your firewall — all traffic now flows through 443 with automatic TLS.Logging in for the first time
Open the admin interface at /ui: LiteLLM displays a username/password form. Enter admin and the password provided, then create your virtual keys from the dashboard.
Supply chain: pin, verify, rotate keys
An LLM gateway holds the keys of all your providers, which is exactly what a credential stealer is after. The report thread opened on the LiteLLM repository (issue 47826) states that the version 1.107.0 wheels published on PyPI carried three files foreign to the project, cook.pth, cook.py and sitecustomize.py, dropped at the root of site-packages: a credential stealer and a code-execution loader, started on every Python launch. PyPI quarantined that version. The maintainers say they removed the malicious Docker images, and their incident write-up describes a hijack of their release pipeline using developer tokens. The original report said it had not inspected the images; a later update in the same thread found the three files in the main image it checked and recommends treating every image of that version as compromised. This guide cannot know what you pulled on a given day: the four checks below settle it on your own host.
Know what is really running on your host
Read the image digest
A tag such as
main-latestsays nothing about what executes. Rundocker compose imagesthendocker image inspect --format '{{index .RepoDigests 0}}' ghcr.io/berriai/litellm:main-latest(replace it with the reference from your compose file). Note thesha256:you get and compare it with the one of a version whose provenance you have verified.Look for the named files, without starting Python
In a suspect environment every Python start replays the malicious code,
pipincluded. Search with a tool that does not launch it:docker compose exec litellm find / -name 'cook.pth' -o -name 'cook.py' -o -name 'sitecustomize.py' -o -name 'litellm-1.107.0.dist-info' 2>/dev/null(if the image shipsfind). A lonesitecustomize.pyelsewhere on the system can be legitimate; what should alarm you is acook.pthbeside it insite-packages, or alitellm-1.107.0.dist-infofolder. Finding these files confirms exposure; finding nothing does not prove the environment is clean.Read the installed version
Once nothing looks suspicious,
pip show litellm(inside the virtualenv of a direct install) ordocker compose exec litellm pip show litellmreports the version. If it is 1.107.0, or if you doubt an image pulled around that time, go straight to key rotation below.Look for execution traces
The report lists a lock file
/tmp/remote_code_loader.lock, an orphanedpythonprocess whose working directory is/, and a marker file/tmp/.pc_once_*written only after a successful upload. Its absence proves nothing: a recreated container wipes/tmp. Use it as a hint, not a verdict.
Pin by digest rather than by tag
A tag is a pointer the publisher can move; the @sha256: digest names exact bytes. In your docker-compose.yml, write image: ghcr.io/berriai/litellm:<version>@sha256:<digest> with an explicit version published before the incident or verified since, never latest, main-latest or main-stable. The maintainers recommend pinning an explicit version, and the issue thread advises the digest. Keep the limit in mind: a digest proves you always pull the same bytes, not who built them. The 1.107.0 images could have been pinned without any alarm. Pinning therefore freezes a decision you made; it does not replace making it.
Update without surprises
- Read the release notes and the security thread before changing version, instead of letting
docker compose pullfollow a tag. - Fetch the new digest from a test machine, start the gateway without real keys, check its behaviour, and only then carry the digest to production.
- When the publisher ships provenance attestations, verify them: they say where an image comes from, which a bare digest does not.
- Keep the previous digest in your configuration repository: rolling back becomes a one-line change.
- After any security alert, rebuild containers and virtual environments from scratch rather than cleaning in place.
If 1.107.0 ever ran on your host: rotate every key
According to the report, if that version started on a host that reaches the Internet, assume the upload succeeded. The code it describes reads the environment variables of readable processes and, when DATABASE_URL is set, extracts the tables where the gateway stores its providers' credentials and models, that is every upstream key it holds. The rotation therefore covers every secret the host could reach, not only the master key. Revoke at the provider first, then generate the replacement key, then review the last month of consumption for usage you do not recognise.
What to renew
- Every provider key stored in the gateway: Anthropic, OpenAI, Azure, Mistral, Bedrock and each of the others.
LITELLM_MASTER_KEYand every virtual key handed to teams.- The database password (
DB_PASSWORDand theDATABASE_URLthat embeds it), as well asUI_PASSWORD. - Any secret in the environment of a process on the host: API tokens, passwords of neighbouring services,
.envfiles. - The SSH keys, Git tokens and cloud credentials reachable from the VPS, if the host carried any.
- After rotation, rebuild the VPS or at least the containers from a verified image: copying fresh secrets into a compromised host exposes them again.
Watch outbound traffic
The report describes an upload to www.litellm.dev, a domain that is not the project's own, and an hourly read of a third-party GitHub repository used as a mailbox for new instructions. The maintainers themselves judge that a network-level restriction is the one that counts, since malicious code ignores application configuration. On a VPS, deny outbound traffic by default and allow only your providers' endpoints; if your firewall filters by address rather than by name, put an outbound proxy with an allowlist in front. Failing that, log the host's DNS lookups and alert on any name the gateway has no reason to call.
Pair LiteLLM with Ollama on the same VPS for a fully air-gapped AI stack. Ollama handles model inference; LiteLLM adds the gateway layer — routing, rate limiting and usage tracking. For privacy-sensitive workloads (legal, finance, medical), this means zero data leaves your infrastructure.
The official documentation
For advanced configuration and options specific to the tool, refer to the official LiteLLM documentation. This guide covers going live on a VPS; the vendor's documentation remains the reference for fine-tuning, major updates and specific use cases.
