A full disk warns nobody
The scenario keeps coming back. A client VPS has run for months, the database writes its logs, backups pile up, and one morning the service stops. No alert went out, and the client finds out before you do. On PostgreSQL, a case reported on the project's official mailing list shows the exact line: PANIC: could not write to file "pg_wal/xlogtemp.11399": No space left on device. A PANIC is not a mere warning: the server can no longer log its writes, it stops, and it only restarts cleanly once space has been freed.
The hard part is not detecting a full disk, it is detecting it beforehand. Between 80% usage and the outage there are often hours or days, and that is exactly the window a well-tuned alert must catch. This guide does not redo the installation of a monitoring stack: it puts the useful minimum on every VPS, in shell, with no second server to maintain.
The six minimum alerts and the heartbeat, on every VPS
- Disk — two thresholds, 80% as a warning and 90% as critical, on every mount point that matters.
- Inodes — a disk can refuse writes while bytes remain free, if inodes are exhausted.
- Memory and OOM — detect that the kernel killed a process, not only that RAM is high.
- systemd services — get notified when the database, web server or proxy enters the failed state.
- Certificates — alert 21 days before expiry, when automatic renewal has failed silently.
- Backups — alert on the absence of a recent success, not only on an explicit error.
- The check itself — an external heartbeat that warns you when alerts stop being sent.
Prerequisites in numbers: almost nothing
Everything below fits in a few dozen lines of POSIX shell. You need a Linux VPS with curl, cron or systemd timers, root access to write in /usr/local/bin and /etc, and about 1 MB of disk. The memory cost is that of a script lasting one second every five minutes: negligible, even on the smallest VPS.
Two external pieces complete the setup: a receiving channel (a Telegram bot, or any webhook reachable over HTTP) and a heartbeat service, which receives a regular ping and warns you when it stops. For that second point, Healthchecks.io offers a free plan limited to 20 jobs and 100 log entries per job, which covers a small fleet comfortably; check those limits at the source before relying on them. If you want no third-party service, Uptime Kuma's push mode plays the same role on a machine of your own.
Disk and inodes: the thresholds and what shifts them
The useful command is df -h --output=pcent,target: it prints only the percentage and the mount point, which is easy to read and parse. For inodes, the same idea becomes df --output=ipcent,target. A sessions folder or a pile of small cache files can use up every inode of an ext4 file system while df -h still shows 40% usage: the write fails all the same with No space left on device.
There is a second subtlety. On ext4, 5% of blocks are reserved by default for privileged processes, according to the tune2fs manual page; that is what lets system daemons keep writing when an ordinary user has filled the disk. Consequence: a service running under an unprivileged account, such as the database, sees df report 100% while the disk is about 95% used. Hence the thresholds chosen: 80% to have time to act, 90% to act immediately. Waiting for 100% means alerting after the outage.
Set it all up in one hour
Create the alert channel
Create a Telegram bot with BotFather, note its token and the identifier of the conversation that should receive messages. Every Bot API request follows the form
https://api.telegram.org/bot<token>/METHOD_NAME, and the sending method issendMessage. If you prefer a webhook (Slack, Mattermost, ntfy), only the last line of the sending script changes.Write the environment file
sudo mkdir -p /var/lib/alerte-vpssudo install -m 600 /dev/null /etc/alerte-vps.env
Then add four lines to it:TG_TOKEN=...,TG_CHAT=...,HC_URL=https://hc-ping.com/<uuid>(five-minute check) andHC_URL_JOUR=https://hc-ping.com/<other-uuid>(daily check). Mode 600 keeps the token readable by root only.Write the sending script with repeat suppression
Save these lines in
/usr/local/bin/alerte-envoithen make the file executable withsudo chmod 755. The per-message lock file avoids being woken every five minutes for the same alert: an identical message is only sent again after one hour.#!/bin/sh. /etc/alerte-vps.envf=/var/lib/alerte-vps/$(echo "$1" | md5sum | cut -c1-12)[ -n "$(find "$f" -mmin -60 2>/dev/null)" ] && exit 0touch "$f"curl -fsS -m 10 --retry 3 "https://api.telegram.org/bot${TG_TOKEN}/sendMessage" -d chat_id="${TG_CHAT}" --data-urlencode text="$(hostname) $1"Check disk and inodes
In
/usr/local/bin/alerte-rapide(shebang#!/bin/sh), add the two loops below. Thegrep -vfilter skips file systems that expose no inodes and print a dash.df --output=pcent,target -x tmpfs -x devtmpfs | tail -n +2 | while read p m; do n=${p%\%}; if [ "$n" -ge 90 ]; then alerte-envoi "CRIT disk $m"; elif [ "$n" -ge 80 ]; then alerte-envoi "WARN disk $m"; fi; donedf --output=ipcent,target -x tmpfs -x devtmpfs | tail -n +2 | grep -v '^ *-' | while read p m; do [ "${p%\%}" -ge 90 ] && alerte-envoi "INODES $m"; doneDetect processes killed by the OOM killer
The
oom_killcounter in/proc/vmstatonly grows since boot. It is compared with the previous value, kept in a file; after a reboot the counter restarts from zero, so the comparison is wrong once, then recovers.n=$(awk '/^oom_kill/ {print $2}' /proc/vmstat)old=$(cat /var/lib/alerte-vps/oom 2>/dev/null || echo 0)[ "$n" -gt "$old" ] && alerte-envoi "OOM kill"echo "$n" > /var/lib/alerte-vps/oomAdd the heartbeat and schedule it
End
alerte-rapidewith the ping. It is only reached if the script ran to the end; if cron stops, if the disk goes read-only or if the script crashes, the ping stops and the external service warns you. Then add the cron lines in/etc/cron.d/alerte-vps.. /etc/alerte-vps.env; curl -fsS -m 10 --retry 5 -o /dev/null "$HC_URL"PATH=/usr/local/bin:/usr/bin:/bin*/5 * * * * root /usr/local/bin/alerte-rapide30 7 * * * root /usr/local/bin/alerte-quotidienCheck certificates and backups once a day
In
/usr/local/bin/alerte-quotidien, add these lines. The-checkendoption ofopenssl x509fails if the certificate expires within the given number of seconds; 1814400 seconds equal 21 days. The backup test alerts when no non-empty.gzfile is newer than 1560 minutes, that is 26 hours. The script ends with its own ping, to the second check: without it, a daily check that fell silent would go unnoticed.echo | openssl s_client -servername your-domain.com -connect your-domain.com:443 2>/dev/null | openssl x509 -noout -checkend 1814400 || alerte-envoi "CERT your-domain.com"find /var/backups/app -name '*.gz' -mmin -1560 -size +1k | grep -q . || alerte-envoi "BACKUP absent". /etc/alerte-vps.env; curl -fsS -m 10 --retry 5 -o /dev/null "$HC_URL_JOUR"Hook the critical services to OnFailure
Create
/etc/systemd/system/[email protected]with the content below, thensudo systemctl daemon-reload.[Unit]Description=Alerte echec de %i[Service]Type=oneshotExecStart=/usr/local/bin/alerte-envoi "FAILED %i"
For each service to watch, open a drop-in withsudo systemctl edit mariadb.serviceand add these two lines.[Unit]OnFailure=notify-failure@%n.serviceForce an alert to test, then clean up
An alert that never fired is not an alert. First check the free space, create a file big enough to cross the threshold, run the script by hand, then delete the file. If the message does not arrive, the defect is in the channel, and it is better to find out now.
df -h --output=pcent,target /var/tmpsudo fallocate -l 2G /var/tmp/test-alerte.imgsudo /usr/local/bin/alerte-rapidesudo rm /var/tmp/test-alerte.img
Memory: kernel OOM or cgroup limit
Two mechanisms kill a process for lack of memory, and they are not read in the same place. The first is the kernel OOM killer, when the whole machine runs short of RAM: the kernel log then holds a message like Out of memory: Killed process, which journalctl -k lets you find. The second is a cgroup limit. According to the systemd documentation, when a unit's usage cannot be contained under MemoryMax=, the OOM killer is invoked inside the unit: the rest of the machine may have free memory.
The oom_kill counter in /proc/vmstat covers the first case at machine scale. For a specific unit, the memory.events file of its cgroup exposes its own oom_kill counter, defined by the kernel documentation as the number of processes of the cgroup killed by any kind of OOM killer. On a VPS that hosts a hungry application, such as the one described in the guide on n8n and OOM in production, this second counter explains the stops that free -h never shows.
OnFailure: what it covers and what it does not
According to the systemd documentation, OnFailure= lists the units activated when the unit enters the failed state. The [email protected] template receives the faulty unit's name through the %n specifier, and the alert message can attach the output of systemctl status or journalctl -u for context.
The limit is structural: a service that runs but no longer answers never went through the failed state, so OnFailure= says nothing. A service restarted by Restart= only gets there once the start limit is reached. For those cases, add a simple application probe, for example a curl on the application's health address, in the fast script. The syntax of timers and cron is covered in the cron and systemd timers guide, and there is no point repeating it here.
Certificates and backups: alert on the absence of success
An automatic certificate renewal can fail silently for weeks: a modified DNS record, a blocked port, a broken plugin. The certificate does not expire that day, it expires ninety days later, and nobody remembers. The openssl test queries what the visitor actually receives, which beats reading a file on disk. On a server managed by Certbot, certbot certificates lists the certificates and their expiry, and certbot renew --dry-run checks that the next renewal will succeed.
Same logic for backups. An alert triggered by the failure of a backup script does not see the case where the script never ran. The correct alert sits on the absence: no recent file, or a zero-size file. That is what the find -mmin line of the daily script does, and why it stays quiet when all is well and talks as soon as the backup chain stops, whatever the reason.
Home-made scripts or a monitoring stack
Scroll the table
| Criterion | Shell scripts and OnFailure | Beszel or Uptime Kuma |
|---|---|---|
| Setup | One hour, a few files in `/usr/local/bin` | A hub to host, agents to install on every VPS |
| Target fleet | 1 to 5 VPS, up to 10 if thresholds are identical | Beyond 5 to 10 VPS, or as soon as comparison is needed |
| History and charts | None, only the moment of the alert | Charts, trends, comparison between machines |
| Maintenance | Nothing to update outside the shell | A second server and its updates to maintain |
| Application services | A `curl` probe to write yourself | HTTP and TCP probes and status page ready to use |
| Monitor failure | Detected by the external heartbeat | To watch separately, the hub is itself a machine |
| When to choose it | Minimum alerts, small fleet, no extra server | History wanted, growing fleet, shared team |
Watch the watcher
The trap of these scripts is to become a false assurance: they stay silent because all is well, or because they are broken, and silence looks the same in both cases. The heartbeat ping solves it. Set the check's expected period slightly above the cron interval and add a grace margin: a silence alert arriving twenty minutes late beats a false positive at every reboot. Also keep /etc/alerte-vps.env in mode 600 and never commit the token.
Troubleshooting: the errors that keep coming back
The alerts themselves fail in predictable ways. Here are the cases met most often, with the message or symptom to recognise and the remedy. Each one is checked by hand before looking further.
Messages and symptoms with their remedy
PANIC: could not write to file "pg_wal/...": No space left on device— the volume holding PostgreSQL's logs is full. First free space outside PostgreSQL (rotated logs, old backups), never deletepg_walfiles by hand, then restart the service.No space left on devicewhiledf -hshows free space — inodes are exhausted. Rundf --output=ipcent,target, find the folder holding millions of small files (sessions, cache) and purge it.- Disk full but
ducannot find the space — deleted files are still held open by a process.sudo lsof +L1lists them; restarting the process frees the space. oom_killrises, nothing injournalctl -k— the kernel log rolled over, or the kill comes from aMemoryMax=limit. Readmemory.eventsin the unit's cgroup directory andjournalctl -ufor the unit.- A unit runs but no longer answers, no alert — that is the limit of
OnFailure=. Add acurlprobe on the health address and alert on the return code. - No Telegram message, no visible error — cron mails the script output locally, where nobody reads it. Run
alerte-envoi testby hand withsudoand read thecurlresponse. - The certificate nears expiry although Certbot is installed — run
certbot renew --dry-runand read the validation error, then fix the DNS or port involved.
The official documentation
Every behaviour described here comes from a primary source. For OnFailure=, the systemd.unit manual page; for MemoryMax=, systemd.resource-control; for the oom and oom_kill counters, the kernel's cgroup v2 documentation; for the reserved 5%, the tune2fs manual page. On the sending side, the Telegram Bot API describes the request form, and the Healthchecks API documentation the ping addresses, with the pricing page for the free plan limits. For renewal, the Certbot documentation. The PostgreSQL message quoted comes from a thread on the project's official mailing list.
Simple alerts, the same on every VPS
Six alerts and a heartbeat do not replace a monitoring stack, they cover what really brings a server down: disk, inodes, memory, services, certificates, backups. The main gain is uniformity: same thresholds, same messages, same test gesture on every machine of the fleet. When the fleet grows or history becomes necessary, the move to a stack is prepared in the Beszel guide and in the proactive alerts guide with Beszel and Uptime Kuma, which cover the installation end to end.
