Why self-host Jan on a VPS
Jan works with a local inference engine (Cortex/llama.cpp) that loads GGUF models directly on the machine. On a personal workstation, it is single-user; on a VPS, you expose the Jan inference server via its OpenAI-compatible API on port 1337, which allows your applications, scripts and colleagues to consume the same private model. No data goes to a third-party provider: prompts and completions stay in the VPS memory. It is the ideal option when confidentiality is non-negotiable or when you want to eliminate any API bill by leveraging open-weights models.
Concrete benefits of self-hosting
- 100% local inference: no prompt is sent to an external service
- OpenAI-compatible API on port 1337, connectable from any existing client
- Zero API cost: only the VPS resources matter, not a token quota
- Centralization of GGUF models on a shared server rather than one download per workstation
- Control of model versions and reproducibility of responses for your tests
- Ability to serve several internal applications behind a single instance
Hardware and software requirements
Jan runs models on the CPU by default, and RAM is the determining factor. A quantized 7B model in Q4 requires about 6 to 8 GB of RAM to run comfortably; so aim for a VPS with 8 GB of RAM and 4 vCPU for acceptable response times, plus 20 to 40 GB of disk depending on the number of GGUF models stored. For 13B or larger models, go up to 16 GB. On the software side: Docker and Docker Compose, a domain pointing to the VPS, and a reverse proxy with authentication, because the Jan API does not enforce a key by default.
Headless deployment of Jan
Provision the VPS
Choose an offer with enough RAM for your models. Install Docker, create /opt/jan/models which will serve as a persistent volume for the GGUF files so as not to re-download them at each update.
Launch the Jan server in a container
Start the Jan server image by mounting the models volume and exposing port 1337 only on the local interface: docker run -d -p 127.0.0.1:1337:1337 -v /opt/jan/models:/root/jan/models --name jan menloltd/cortex. Restricting to 127.0.0.1 avoids any direct exposure.
Download a model
Via the API or the Cortex CLI, retrieve a model, for example cortex pull llama3.1:8b-gguf. Check that it loads correctly with a test request on http://127.0.0.1:1337/v1/models.
Test the OpenAI-compatible API
Send a test completion: curl http://127.0.0.1:1337/v1/chat/completions -H 'Content-Type: application/json' -d '{"model":"llama3.1:8b","messages":[{"role":"user","content":"Bonjour"}]}' and confirm the response.
Secure with a reverse proxy and a key
Place Caddy or Nginx in front of port 1337, enable Let's Encrypt HTTPS and add an authentication layer (basic auth or a verified Authorization header) to prohibit any anonymous access to your inference server.
Expose the domain
Publish https://ia.yourdomain.com/v1 and reconfigure your clients by pointing their base_url to this address. Your applications now consume Jan as if it were OpenAI.
Without a GPU, throughput in tokens/second depends heavily on the number of CPU threads. Explicitly set --n-threads to the number of real vCPU of the VPS and favor Q4_K_M quantizations: the quality/speed trade-off is optimal for a CPU server and halves the memory footprint compared to a non-quantized model.
