Why self-host Whisper on a VPS
Cloud transcription APIs charge per minute of audio and require sending your recordings to external servers — a problem as soon as it involves confidential meetings, HR interviews or medical data. Self-hosting Whisper on a VPS gives you a private transcription endpoint, available 24/7, that processes your files locally. With the faster-whisper implementation (based on CTranslate2), you get transcriptions up to 4 times faster than the original implementation, even on CPU. You choose the model size (tiny to large-v3) according to the speed/accuracy trade-off you need, and you plug it all into your own applications via a REST API.
The concrete benefits of self-hosting
- Privacy: your sensitive audio never leaves your VPS.
- Multilingual transcription and translation into English included in the same model.
- Word- or segment-level timestamping, ideal for generating SRT/VTT subtitles.
- No duration limit per file, unlike online APIs.
- Fixed cost instead of billing per minute of transcribed audio.
- Integration via REST API into your internal tools (CRM, podcast, customer support).
Hardware and software prerequisites
Whisper is demanding mainly in RAM and compute. On CPU, the base or small model runs comfortably with 4 GB of RAM and 2 vCPUs; the large-v3 model rather requires 8 to 10 GB of RAM and 4 vCPUs to stay usable. A GPU VPS (6 GB of VRAM) hugely accelerates large-v3. Plan for a few GB of disk for the weights (the large-v3 weighs ~3 GB) and temporary space for the uploaded audio files. On the software side: Ubuntu 22.04, Docker, ffmpeg (essential for decoding audio formats), and a subdomain like whisper.mydomain.com.
Step-by-step deployment
Prepare the environment
Via SSH, install Docker and check for the presence of ffmpeg (apt install ffmpeg). Create a whisper/ folder that will contain your docker-compose.yml and an ./audio volume for the files to process.
Choose an image with an API
Use an image that directly exposes a REST API, for example onerahmet/openai-whisper-asr-webservice. In the compose, set ASR_ENGINE=faster_whisper and ASR_MODEL=small (adjust according to your RAM), and map port 9000.
Start the service
Start with docker compose up -d. On the first call, the container automatically downloads the model weights; mount a volume on /root/.cache so as not to re-download it on every restart.
Test the transcription
Send a file locally: curl -F "[email protected]" "http://localhost:9000/asr?output=txt". You should get the transcription back within a few seconds to a few minutes depending on the model.
Expose over HTTPS with authentication
Place Caddy or Nginx in front of port 9000, with authentication (an API key in a header or Basic Auth) to prevent a third party from using your endpoint. Caddy will handle the Let's Encrypt certificate automatically for whisper.mydomain.com.
Wire up your workflows
From your applications, call https://whisper.mydomain.com/asr?output=srt to obtain subtitles directly, or output=json for the timestamped segments to reinject into your database.
To process long recordings on CPU without saturating the RAM, enable faster-whisper's VAD (voice activity detection) to skip silences, and set compute_type=int8: accuracy barely drops but memory consumption falls sharply, which lets you run medium where large would not fit. For large volumes, set up a queue (Redis + worker) rather than synchronous calls.
