[{"data":1,"prerenderedAt":207},["ShallowReactive",2],{"seo-verification":3,"blog-installing-apache-airflow-on-a-vps-self-hosted-automation-ma-en":6,"promo-bar-likely":206},{"google":4,"bing":5},"EycwPY2XMyTkVzas3n1ygeNJFGAH513qrMjfDljzsMQ","",{"key":7,"data":8},"blog-installing-apache-airflow-on-a-vps-self-hosted-automation-ma-en",{"id":9,"slug":10,"slugs":11,"title":15,"excerpt":16,"readTime":17,"views":18,"isPinned":19,"noindex":19,"publishedAt":20,"updatedAt":21,"category":22,"categories":27,"featuredImage":29,"bgImage":30,"posterImage":31,"relatedSolution":32,"intro":35,"sections":36,"ctaTitle":148,"ctaBody":149,"ctaButton":150,"ctaUrl":151,"relatedPosts":152},31,"installing-apache-airflow-on-a-vps-self-hosted-automation",{"fr":12,"en":10,"ar":13,"es":14},"installer-apache-airflow-vps","تثبيت-apache-airflow-على-خادم-vps-أتمتة-ذاتية-الاستضافة","instalar-apache-airflow-en-un-vps","Install Apache Airflow on VPS: Docker Compose and PostgreSQL","Install Apache Airflow on VPS with Docker Compose: PostgreSQL as metadata backend, multi-worker CeleryExecutor, encrypted connections and debugging — complete 2026 guide.",12,1,false,"2026-05-20T00:00:00+00:00","2026-11-04T16:30:25+00:00",{"id":23,"name":24,"slug":25,"color":26,"icon":25},2,"Automation","automatisation","bg-brand-action\u002F10 text-brand-action",[28],{"id":23,"name":24,"slug":25,"color":26,"icon":25},null,"\u002Fblog\u002Fcovers\u002Fbg.svg","\u002Fblog\u002Fcovers\u002Finstaller-apache-airflow-vps-poster.svg",{"categorySlug":33,"appSlug":34},"automation-workflows","airflow","Apache Airflow is the open-source standard for orchestrating data pipelines, ETL workflows and ML tasks in Python. Self-hosted on a VPS, it frees you from the cost of a managed service (Cloud Composer, MWAA) while keeping full control over your data and dependencies. This guide covers everything from zero to a production-ready setup: PostgreSQL as metadata backend (never SQLite on Docker volumes), CeleryExecutor for task distribution, encrypted secrets and connections, and the right reflexes for fast debugging.",[37,41,51,54,57,79,82,85,88,100,103,137,141],{"type":38,"title":39,"body":40},"h2","Why host Apache Airflow on your own VPS","Cloud orchestrators like Cloud Composer (Google) or MWAA (Amazon) bundle Airflow as a managed service, but they bill the environment by the hour — regardless of your actual workload. Self-hosting on a VPS means you only pay for the server, and your DAGs run close to your internal data sources without external network transit. The freedom goes further: you choose the Airflow version, Python providers, database connections, and executor (Local, Celery, Kubernetes). For pipelines processing sensitive data or regulated environments, staying on-premise with a dedicated VPS is not an option but a requirement. Cost-wise, a 4 vCPU \u002F 8 GB RAM VPS is a fraction of a managed environment for the same daily workload.",{"type":42,"title":43,"items":44},"ul","What Airflow orchestrates in practice",[45,46,47,48,49,50],"**Data pipelines and ETL** — extract from APIs or databases, transform and load into a data warehouse.","**ML workflows** — model training, evaluation, automated deployment with step dependencies.","**Complex scheduled tasks** — backfills, catchups, per-task configurable retry on failure.","**Third-party integrations** — 80+ official providers: PostgreSQL, MySQL, S3, BigQuery, dbt, Spark, Kubernetes, HTTP.","**Reporting pipelines** — automated periodic report generation and delivery via email or Slack.","**Microservice orchestration** — trigger remote jobs via API and wait for their result before proceeding.",{"type":38,"title":52,"body":53},"VPS requirements and technical stack","Airflow is the most resource-intensive stack in this category in CeleryExecutor mode, as it runs multiple services in parallel. **For production**: minimum 4 vCPU and 8 GB RAM — start with a Power VPS or a dedicated VPS. **For development or testing**: 2 vCPU and 4 GB RAM are enough in LocalExecutor mode (no Celery or Redis). Plan for 40 GB SSD for logs and metadata. Software requirements: Docker 24+ and Docker Compose v2 (check with `docker compose version`, not legacy `docker-compose` v1), Ubuntu 22.04 LTS or Debian 12, and a domain name pointing to your VPS for the HTTPS reverse proxy.",{"type":38,"title":55,"body":56},"Airflow Docker Compose architecture","Airflow's official `docker-compose.yaml` deploys six services forming a complete architecture. The **webserver** serves the graphical interface on port 8080: DAG visualization, manual triggering, log inspection. The **scheduler** is the system's core: it parses DAGs, schedules tasks according to their schedule and dependencies, and queues them for workers. The **worker** executes tasks assigned by the scheduler — you can run multiple workers for parallelization. The **triggerer** handles deferrable tasks (async sensors since Airflow 2.2+) without blocking worker slots. **PostgreSQL** is Airflow's metadata database: DAG run state, structured logs, XComs, variables and connections — it is the most critical component. **Redis** serves as the message broker between the scheduler and Celery workers: each ready task transits through a Redis queue before being picked up by an available worker.",{"type":58,"title":59,"steps":60},"steps","Deploy Airflow 3.x with Docker Compose",[61,64,67,70,73,76],{"title":62,"body":63},"Fetch the official docker-compose and create directories","Create the working directory, download the reference file and prepare the volume-mounted folders:\n```bash\nmkdir -p \u002Fopt\u002Fairflow && cd \u002Fopt\u002Fairflow\ncurl -LfO 'https:\u002F\u002Fairflow.apache.org\u002Fdocs\u002Fapache-airflow\u002Fstable\u002Fdocker-compose.yaml'\nmkdir -p .\u002Fdags .\u002Flogs .\u002Fplugins .\u002Fconfig\n```\nThese directories must exist before starting the containers. If Docker creates them, it assigns `root` ownership and Airflow cannot write to them.",{"title":65,"body":66},"Configure the .env file (UID, PostgreSQL, secret key)","Create `\u002Fopt\u002Fairflow\u002F.env` with the essential variables:\n```bash\necho \"AIRFLOW_UID=$(id -u)\" > .env\necho \"AIRFLOW__DATABASE__SQL_ALCHEMY_CONN=postgresql+psycopg2:\u002F\u002Fairflow:airflow@postgres\u002Fairflow\" >> .env\necho \"AIRFLOW__CORE__FERNET_KEY=$(python3 -c 'from cryptography.fernet import Fernet; print(Fernet.generate_key().decode())')\" >> .env\n```\n`AIRFLOW_UID` aligns volume permissions with your system user. `AIRFLOW__DATABASE__SQL_ALCHEMY_CONN` forces PostgreSQL — making it explicit in `.env` avoids any ambiguity. `FERNET_KEY` encrypts connections and variables stored in the database — **generate it once and back it up**: if you lose it, all your secret connections become unreadable.",{"title":68,"body":69},"Initialize the database","Apply migrations and create the first admin account:\n```bash\ndocker compose up airflow-init\n```\nThe `airflow-init` container connects to PostgreSQL, applies all schema migrations and creates the `airflow` user (password `airflow`). Wait for exit code `0` — any other code indicates a PostgreSQL connection issue (not yet healthy) or a missing image. **Change the password on first login** via the UI or `docker compose exec airflow-webserver airflow users reset-password`.",{"title":71,"body":72},"Start all services","Launch the full stack in the background:\n```bash\ndocker compose up -d\ndocker compose ps\n```\nAll containers should reach `healthy` state within one to three minutes. If a service stays in `starting`, check its logs:\n```bash\ndocker compose logs airflow-scheduler --tail=50\n```\nThe webserver is accessible at `http:\u002F\u002Flocalhost:8080`. Do not expose this port publicly — use a reverse proxy.",{"title":74,"body":75},"Create a custom admin account","The default `airflow\u002Fairflow` account created by `airflow-init` is for first-boot only. Create your permanent account:\n```bash\ndocker compose exec airflow-webserver airflow users create \\\n  --username admin \\\n  --firstname Firstname \\\n  --lastname Lastname \\\n  --role Admin \\\n  --email admin@yourdomain.com \\\n  --password YourStrongPassword\n```\nThen delete the default account via Admin → Users in the web interface.",{"title":77,"body":78},"Expose Airflow behind an HTTPS reverse proxy","Never expose port 8080 directly — the Airflow UI displays connections, variables and sensitive logs. With Caddy (recommended for automatic TLS):\n```bash\napt install -y caddy\ncat > \u002Fetc\u002Fcaddy\u002FCaddyfile \u003C\u003C 'EOF'\nairflow.yourdomain.com {\n    reverse_proxy localhost:8080\n}\nEOF\nsystemctl reload caddy\nufw deny 8080\n```\nCaddy automatically obtains and renews the Let's Encrypt certificate. With nginx, create a standard vhost with `proxy_pass http:\u002F\u002F127.0.0.1:8080` and block the port with `ufw deny 8080`.",{"type":38,"title":80,"body":81},"Configure PostgreSQL as the metadata backend","**Never use SQLite in production with Airflow on Docker.** SQLite is not designed for concurrent multi-process access: the scheduler, webserver, and workers access the database simultaneously, and SQLite on Docker volumes has locking limitations that cause corruption and data loss — the Airflow project explicitly documents this warning since version 2.0. The official `docker-compose.yaml` already includes a PostgreSQL 13 service, but verify your `.env` explicitly forces the connection:\n```\nAIRFLOW__DATABASE__SQL_ALCHEMY_CONN=postgresql+psycopg2:\u002F\u002Fairflow:airflow@postgres\u002Fairflow\n```\nFor production, customize the PostgreSQL password. In `docker-compose.yaml`, find the `postgres` section and modify `POSTGRES_PASSWORD`. Update `AIRFLOW__DATABASE__SQL_ALCHEMY_CONN` accordingly. Back up the PostgreSQL database regularly: `docker compose exec postgres pg_dump -U airflow airflow > backup_$(date +%Y%m%d).sql`. If you already manage an external PostgreSQL server, remove the `postgres` service from the compose and point `SQL_ALCHEMY_CONN` to your instance — this is the recommended configuration for multi-VPS setups.",{"type":38,"title":83,"body":84},"Switch to CeleryExecutor for multiple workers","The **LocalExecutor** runs tasks as subprocess of the scheduler — simple, but limited to a single node. The **CeleryExecutor** decouples execution: the scheduler pushes tasks into a Redis queue, and one or more workers consume them independently. The official `docker-compose.yaml` enables CeleryExecutor by default. To scale out, simply replicate the worker service:\n```bash\ndocker compose up -d --scale airflow-worker=3\n```\nThis starts three workers consuming the same Redis queue in parallel. Each worker handles `parallelism` simultaneous tasks (configurable via `AIRFLOW__CORE__PARALLELISM`, default 32). To distribute workers across multiple VPS, point them all to the same Redis and PostgreSQL — that is standard Celery architecture. Add `AIRFLOW__CELERY__WORKER_CONCURRENCY=8` in `.env` to fine-tune per-worker concurrency based on available resources on each node.",{"type":38,"title":86,"body":87},"Manage connections and secret variables","Airflow encrypts connections and sensitive variables in the database using the `FERNET_KEY` configured in `.env`. **Via the UI**: Admin → Connections → Add Connection — fill in the full URI or individual fields (host, login, password, port). Values are Fernet-encrypted before being written to the database. **Via environment variables** (recommended for CI\u002FCD): an env variable prefixed with `AIRFLOW_CONN_` overrides any connection stored in the database. Example in `.env`:\n```\nAIRFLOW_CONN_MY_POSTGRES=postgresql:\u002F\u002Fuser:password@host:5432\u002Fdbname\n```\nThe connection name in your DAGs is then `my_postgres` (prefix and uppercase are stripped). **Secret Backend**: for Kubernetes or cloud environments, configure `AIRFLOW__SECRETS__BACKEND=airflow.providers.amazon.aws.secrets.secrets_manager.SecretsManagerBackend` (or HashiCorp Vault, GCP Secret Manager) — Airflow will resolve connections from your vault rather than from PostgreSQL. **Variables**: Admin → Variables in the UI, or via env `AIRFLOW_VAR_VARIABLE_NAME`. Avoid storing secrets in unencrypted Variables — prefer Connections or Secret Backend.",{"type":58,"title":89,"steps":90},"Create and test your first Python DAG",[91,94,97],{"title":92,"body":93},"Create the DAG file in the \u002Fdags folder","Create `\u002Fopt\u002Fairflow\u002Fdags\u002Fmy_first_dag.py`:\n```python\nfrom airflow.sdk import DAG, task\nfrom datetime import datetime\n\nwith DAG(\n    dag_id='my_first_dag',\n    schedule='@daily',\n    start_date=datetime(2025, 1, 1),\n    catchup=False,\n    tags=['example'],\n) as dag:\n\n    @task\n    def extract():\n        return {'rows': 42}\n\n    @task\n    def transform(data: dict):\n        print(f\"Processing {data['rows']} rows\")\n        return data['rows'] * 2\n\n    @task\n    def load(result: int):\n        print(f\"Final result: {result}\")\n\n    load(transform(extract()))\n```\nNote the use of `schedule` (not `schedule_interval`, removed in Airflow 3).",{"title":95,"body":96},"Verify the scheduler parses the DAG without errors","The scheduler automatically detects and parses `.py` files in the `dags` folder. Check manually:\n```bash\ndocker compose exec airflow-scheduler airflow dags list | grep my_first_dag\n```\nIf the DAG does not appear after 30 seconds, look for parsing errors:\n```bash\ndocker compose logs airflow-scheduler | grep -i 'error\\|exception' | tail -20\n```\nA Python import error in the DAG file blocks all detection — fix the syntax and the scheduler will automatically retry parsing.",{"title":98,"body":99},"Trigger a manual run and inspect logs","In the Airflow UI (`https:\u002F\u002Fairflow.yourdomain.com`), navigate to DAGs, find `my_first_dag`, enable it (toggle), then click ▶ Trigger DAG. Follow the execution in real time in the Graph or Grid view. Click on a task, then Logs to see the function's standard output. From the terminal:\n```bash\ndocker compose exec airflow-scheduler airflow dags trigger my_first_dag\ndocker compose exec airflow-scheduler airflow dags state my_first_dag $(date +%Y-%m-%dT%H:%M:%S+00:00)\n```",{"type":38,"title":101,"body":102},"Secure your Airflow installation","An Airflow installation potentially exposes your database connections, variables and execution logs. Apply these measures from the start. **Authentication**: Airflow 3 uses FAB (Flask-AppBuilder) with password authentication by default. For an organization, connect Airflow to your directory via LDAP, or enable OAuth2 (GitHub, Google, Azure AD) by configuring the corresponding FAB provider. **Disable example DAGs**: add to `.env`:\n```\nAIRFLOW__CORE__LOAD_EXAMPLES=False\n```\nThese dummy DAGs clutter the UI and can be misleading. **Firewall on port 8080**: `ufw deny 8080` — only the reverse proxy should access it from localhost. **No Fernet key in logs**: never log `os.environ`, which would print the `FERNET_KEY` in plain text. **Secret rotation**: if you need to change the `FERNET_KEY`, use `airflow db rotate-fernet-key` to re-encrypt all stored connections before updating the variable.",{"type":104,"title":105,"headers":106,"rows":111},"comparison","LocalExecutor vs CeleryExecutor vs KubernetesExecutor",[107,108,109,110],"Criterion","LocalExecutor","CeleryExecutor","KubernetesExecutor",[112,117,122,127,132],[113,114,115,116],"Required resources","2 vCPU \u002F 4 GB RAM","4 vCPU \u002F 8 GB RAM + Redis","Kubernetes cluster required",[118,119,120,121],"Parallelism","Limited to scheduler CPU count","Horizontal — as many workers as needed","One pod per task — near-unbounded parallelism",[123,124,125,126],"Task isolation","Shared subprocesses","Separate workers, same image","Isolated pod per task, configurable image",[128,129,130,131],"Operational complexity","Simple — no Redis","Moderate — Redis to maintain","High — Kubernetes required",[133,134,135,136],"Ideal use case","Dev, tests, light pipelines","VPS production, variable load","Cloud-native, strong isolation required",{"type":138,"title":139,"body":140},"tip","Update Airflow without downtime","For a rolling update without interrupting running DAGs: start by updating the image in `docker-compose.yaml` (e.g. `apache\u002Fairflow:3.3.1`). Stop only the webserver and scheduler, apply migrations, then restart service by service. Celery workers continue executing already-started tasks during the migration:\n```bash\ndocker compose stop airflow-webserver airflow-scheduler\ndocker compose pull\ndocker compose up -d airflow-webserver airflow-scheduler\ndocker compose exec airflow-scheduler airflow db upgrade\ndocker compose up -d\n```\nAlways check the official CHANGELOG between minor versions — some schema migrations are irreversible. Keep a fresh PostgreSQL backup before any version upgrade.",{"type":38,"title":142,"body":143,"cta_url":144,"cta_title":145,"cta_body":146,"cta_button":147},"Debug and read Airflow logs","Airflow generates logs at multiple levels and understanding their organization greatly speeds up debugging. **Task logs**: each task writes to `.\u002Flogs\u002Fdag_id\u002Frun_id\u002Ftask_id\u002F`. Accessible from the UI (click on task → Logs) or from the command line:\n```bash\ndocker compose exec airflow-scheduler airflow tasks logs my_first_dag extract 2025-01-01\n```\n**Scheduler logs**: the most important for diagnosing DAGs that do not trigger:\n```bash\ndocker compose logs airflow-scheduler --follow --tail=100\n```\n**Remote logging**: for multi-worker setups or long-retention logs, offload to S3 or GCS:\n```\nAIRFLOW__LOGGING__REMOTE_LOGGING=True\nAIRFLOW__LOGGING__REMOTE_BASE_LOG_FOLDER=s3:\u002F\u002Fmy-bucket\u002Fairflow-logs\nAIRFLOW__LOGGING__REMOTE_LOG_CONN_ID=my_aws\n```\n**Silent errors**: if a DAG disappears from the UI without a message, look for scheduler import errors:\n```bash\ndocker compose exec airflow-webserver airflow dags list-import-errors\n```\n**Log purge**: set `AIRFLOW__LOG_RETENTION_DAYS=30` to avoid disk saturation — the `.\u002Flogs` folder can quickly reach several gigabytes on frequent pipelines.","\u002Fapplications\u002Fautomatisation\u002Fairflow","Deploy Airflow on a ServOrbit VPS","ServOrbit offers Power VPS pre-configured with Docker Compose to launch Airflow in under ten minutes. Choose your worker size — we handle the infrastructure.","Deploy Airflow on VPS","The ideal Cloud VPS for Apache Airflow","Airflow 3 in Celery mode demands RAM and several containers. The ServOrbit Cloud VPS provides the resources, preconfigured Docker and automatic SSL to host your data scheduler self-hosted, at a controlled cost.","Deploy my Cloud VPS","\u002Fapplications\u002Fautomation-workflows\u002Fairflow",[153,172,189],{"id":154,"slug":155,"slugs":156,"title":160,"excerpt":161,"readTime":162,"views":163,"isPinned":19,"noindex":19,"publishedAt":164,"updatedAt":165,"category":166,"categories":167,"featuredImage":29,"bgImage":30,"posterImage":169,"relatedSolution":170},3,"install-n8n-on-vps-with-docker-complete-2026-guide",{"fr":157,"en":155,"ar":158,"es":159},"installer-n8n-vps","تثبيت-n8n-على-vps-مع-docker-دليل-شامل-2026","instalar-n8n-en-vps-con-docker","Install n8n on VPS with Docker: complete 2026 guide","Deploy n8n on VPS with Docker, reverse proxy and HTTPS. Covers V8 crash, 502 nginx, npm migration and execution security (GHSA-vrv8-j27g-g7cr advisory, August 2026).",19,4,"2026-06-05T00:00:00+00:00","2026-11-09T20:55:18+00:00",{"id":23,"name":24,"slug":25,"color":26,"icon":25},[168],{"id":23,"name":24,"slug":25,"color":26,"icon":25},"\u002Fblog\u002Fcovers\u002Finstaller-n8n-vps-poster.svg",{"categorySlug":33,"appSlug":171},"n8n",{"id":173,"slug":174,"slugs":175,"title":179,"excerpt":180,"readTime":163,"views":18,"isPinned":19,"noindex":19,"publishedAt":181,"updatedAt":182,"category":183,"categories":184,"featuredImage":29,"bgImage":30,"posterImage":186,"relatedSolution":187},24,"installing-node-red-on-a-vps-self-hosted-automation",{"fr":176,"en":174,"ar":177,"es":178},"installer-node-red-vps","تثبيت-node-red-على-خادم-vps-أتمتة-ذاتية-الاستضافة","instalar-node-red-en-un-vps","Installing Node-RED on a VPS: self-hosted automation","Install Node-RED on a VPS to automate your flow-based workflows, connect your APIs and IoT devices, all self-hosted and secure.","2026-05-27T00:00:00+00:00","2026-11-04T16:30:24+00:00",{"id":23,"name":24,"slug":25,"color":26,"icon":25},[185],{"id":23,"name":24,"slug":25,"color":26,"icon":25},"\u002Fblog\u002Fcovers\u002Finstaller-node-red-vps-poster.svg",{"categorySlug":33,"appSlug":188},"node-red",{"id":190,"slug":191,"slugs":192,"title":196,"excerpt":197,"readTime":154,"views":198,"isPinned":19,"noindex":19,"publishedAt":199,"updatedAt":182,"category":200,"categories":201,"featuredImage":29,"bgImage":30,"posterImage":203,"relatedSolution":204},25,"installing-huginn-on-a-vps-self-hosted-automation",{"fr":193,"en":191,"ar":194,"es":195},"installer-huginn-vps","تثبيت-huginn-على-خادم-vps-أتمتة-ذاتية-الاستضافة","instalar-huginn-en-un-vps","Installing Huginn on a VPS: self-hosted automation","Install Huginn on a VPS to build agents that monitor the web, scrape pages and trigger actions, 100% self-hosted.",0,"2026-05-26T00:00:00+00:00",{"id":23,"name":24,"slug":25,"color":26,"icon":25},[202],{"id":23,"name":24,"slug":25,"color":26,"icon":25},"\u002Fblog\u002Fcovers\u002Finstaller-huginn-vps-poster.svg",{"categorySlug":33,"appSlug":205},"huginn",true,1794270434585]