Why self-host Apache Airflow on a VPS
Airflow is the tool data teams choose when they want to describe their pipelines in pure Python: each DAG is a .py file that defines tasks, their dependencies and their scheduling. The provider ecosystem is immense (SQL databases, S3, BigQuery, dbt, Spark…), and the community gigantic. Managed services (Cloud Composer, MWAA) charge a high hourly rate for the environment; by self-hosting Airflow on a VPS, you get the same engine for the cost of a server, and your DAGs run as close as possible to your internal data sources. It is also a matter of control: provider versions, custom Python dependencies, variables and connections, everything stays in your hands. The official Docker Compose deployment (CeleryExecutor) faithfully reproduces a production architecture: webserver, scheduler, workers, Redis and PostgreSQL.
Concrete benefits of self-hosting
- DAGs in pure Python, versioned in Git, with controlled dependencies and providers.
- Huge ecosystem of operators: SQL, cloud, dbt, Spark, HTTP, and more.
- Robust scheduler: cron, dependencies between tasks, backfill and catchup.
- Scalable Celery workers to absorb hundreds of tasks in parallel.
- Execution close to your internal databases, with no transit through a third-party cloud.
- The cost of a VPS instead of a managed environment billed by the hour.
Technical requirements
Airflow is the most resource-hungry tool in this series in a CeleryExecutor configuration, because it simultaneously runs webserver, scheduler, worker(s), triggerer, Redis and PostgreSQL. The official documentation recommends at least 4 GB of RAM, but for comfortable production aim for 4 vCPU and 8 GB of RAM; 16 GB if your DAGs load pandas or large volumes. Plan for 40 GB of disk, Docker and Docker Compose, and a domain (airflow.yourdomain.com). The LocalExecutor mode lightens the stack (without Celery or Redis) and can run on 4 GB if your needs remain modest.
Deploy Apache Airflow with Docker Compose
Fetch the official compose
Create /opt/airflow, then download the reference file: curl -LfO https://airflow.apache.org/docs/apache-airflow/stable/docker-compose.yaml. Create the expected subfolders: mkdir -p ./dags ./logs ./plugins ./config.
Set the permissions and the UID
Airflow requires a consistent UID for the mounted volumes. Generate the environment file: echo -e "AIRFLOW_UID=$(id -u)" > .env. Without this, the scheduler will not be able to write its logs and will crash on startup.
Initialize the database
Launch the migration and the creation of the admin account: docker compose up airflow-init. This command prepares PostgreSQL, applies the migrations and creates the default user airflow / airflow, to be changed afterwards.
Start the complete stack
Launch all the services: docker compose up -d. Check that webserver, scheduler, worker and triggerer are healthy with docker compose ps. The webserver listens on port 8080.
Reverse proxy and HTTPS
Put Caddy in front of the webserver: airflow.yourdomain.com { reverse_proxy localhost:8080 }. HTTPS protects access to the Airflow UI, which exposes connections, variables and potentially sensitive logs.
Drop in a DAG and run it
Place a minimal hello.py file in ./dags (a DAG with a BashOperator or PythonOperator). The scheduler detects it within a few seconds. Enable it in the UI, trigger a manual execution and inspect the task logs.
Never put heavy logic directly in the DAG file: the scheduler parses all the .py files at regular intervals (min_file_process_interval), and a costly import at module level slows down the entire scheduler. Keep the parsing code light and offload the work into the tasks. To manage custom Python dependencies, build your own image derived from apache/airflow with a requirements.txt rather than installing packages at runtime: your workers start faster and remain reproducible. Finally, monitor the growth of the ./logs folder and purge it regularly.
