Why self-host Apache Airflow on a VPS
Airflow is the tool data teams choose when they want to describe their pipelines in pure Python: each DAG is a .py file that defines tasks, their dependencies and their scheduling. The provider ecosystem is immense (SQL databases, S3, BigQuery, dbt, Spark…), and the community gigantic. Managed services (Cloud Composer, MWAA) charge a high hourly rate for the environment; by self-hosting Airflow on a VPS, you get the same engine for the cost of a server, and your DAGs run as close as possible to your internal data sources. It is also a matter of control: provider versions, custom Python dependencies, variables and connections, everything stays in your hands. The official Docker Compose deployment (CeleryExecutor) faithfully reproduces a production architecture: webserver, scheduler, workers, Redis and PostgreSQL.
Concrete benefits of self-hosting
- DAGs in pure Python, versioned in Git, with controlled dependencies and providers.
- Huge ecosystem of operators: SQL, cloud, dbt, Spark, HTTP, and more.
- Robust scheduler: cron, dependencies between tasks, backfill and catchup.
- Scalable Celery workers to absorb hundreds of tasks in parallel.
- Execution close to your internal databases, with no transit through a third-party cloud.
- The cost of a VPS instead of a managed environment billed by the hour.
Technical requirements
Airflow is the most resource-hungry tool in this series in a CeleryExecutor configuration, because it simultaneously runs webserver, scheduler, worker(s), triggerer, Redis and PostgreSQL. The official documentation recommends at least 4 GB of RAM, but for comfortable production aim for 4 vCPU and 8 GB of RAM; 16 GB if your DAGs load pandas or large volumes. Plan for 40 GB of disk, Docker and Docker Compose, and a domain (airflow.yourdomain.com). The LocalExecutor mode lightens the stack (without Celery or Redis) and can run on 4 GB if your needs remain modest.
Deploy Apache Airflow with Docker Compose
Fetch the official compose
Create /opt/airflow, then download the reference file: curl -LfO https://airflow.apache.org/docs/apache-airflow/stable/docker-compose.yaml. Create the expected subfolders: mkdir -p ./dags ./logs ./plugins ./config.
Set the permissions and the UID
Airflow requires a consistent UID for the mounted volumes. Generate the environment file: echo -e "AIRFLOW_UID=$(id -u)" > .env. Without this, the scheduler will not be able to write its logs and will crash on startup.
Initialize the database
Launch the migration and the creation of the admin account: docker compose up airflow-init. This command prepares PostgreSQL, applies the migrations and creates the default user airflow / airflow, to be changed afterwards.
Start the complete stack
Launch all the services: docker compose up -d. Check that webserver, scheduler, worker and triggerer are healthy with docker compose ps. The webserver listens on port 8080.
Reverse proxy and HTTPS
Put Caddy in front of the webserver: airflow.yourdomain.com { reverse_proxy localhost:8080 }. HTTPS protects access to the Airflow UI, which exposes connections, variables and potentially sensitive logs.
Drop in a DAG and run it
Place a minimal hello.py file in ./dags (a DAG with a BashOperator or PythonOperator). The scheduler detects it within a few seconds. Enable it in the UI, trigger a manual execution and inspect the task logs.
Never put heavy logic directly in the DAG file: the scheduler parses all the .py files at regular intervals (min_file_process_interval), and a costly import at module level slows down the entire scheduler. Keep the parsing code light and offload the work into the tasks. To manage custom Python dependencies, build your own image derived from apache/airflow with a requirements.txt rather than installing packages at runtime: your workers start faster and remain reproducible. Finally, monitor the growth of the ./logs folder and purge it regularly.
CVE-2026-58076: critical vulnerability and update to Airflow 2.10.4
CVE-2026-58076 (CVSS 9.1) is a remote code execution vulnerability in the DAG serialization component of Apache Airflow, affecting versions prior to 2.10.4. An authenticated attacker can trigger the execution of an arbitrary object during deserialization by injecting a payload into a DAG's metadata. The fix is available in Airflow 2.10.4. To update your Docker Compose instance: change the tag in your docker-compose.yml to apache/airflow:2.10.4, then run docker compose pull && docker compose up -d. Verify the version with docker compose exec airflow-webserver airflow version. Migration tables are updated automatically on first startup. If web access to DAG serialization needs to be disabled while awaiting the update, set AIRFLOW__CORE__DAG_SERIALIZATION=True (enabled by default since 2.1.0) and block access to the worker from outside the Docker network.
Troubleshooting: three common errors
Port 8080 already in use: if docker compose up -d fails with port is already allocated, change the port in docker-compose.yml or stop the service occupying it (lsof -i :8080). Scheduler in running state but not executing DAGs: verify that the ./dags directory is properly mounted (docker compose config | grep dags) and that the DAG .py files are syntactically valid (python3 -c "import dag_file"). DAG not visible in the interface after adding it: the DAG folder scan interval is 30 seconds by default; force a reload with docker compose restart airflow-scheduler. For files modified after startup, the Scheduler picks them up automatically in its next cycle.
