OpenVPN on VPS: install, harden and troubleshoot

Shared commercial VPN solutions require trusting a third-party provider that can log, resell or expose your connection data. By hosting OpenVPN Server on your own VPS, you are the sole administrator: you choose the level of logging, the authorized users and the encryption protocols (TLS 1.3, AES-256-GCM). It is also the ideal solution to interconnect distributed teams, secure access to your internal services or bypass the network restrictions of a corporate environment, all at a controlled cost.

Before starting, check these points on your VPS.

Minimum resources: 1 vCPU and 512 MB of RAM are enough for personal use or up to 10 simultaneous users. Beyond 50 active connections, plan for 2 vCPU and 1 GB of RAM to absorb encryption without bandwidth degradation.

Operating system: Ubuntu 22.04 LTS or Debian 12 (both supported by the official OpenVPN repository). AlmaLinux 9 and Rocky Linux 9 also work with EPEL packages.

Network port: UDP 1194 (default) or TCP 443 to pass through restrictive firewalls. On your ServOrbit VPS, open this port in the security rules before starting the installation. A single port is enough.

Root access: required to create virtual network interfaces (tun0), modify iptables rules and enable IP forwarding.

IP forwarding: enable it before continuing:

echo 'net.ipv4.ip_forward=1' >> /etc/sysctl.conf
sysctl -p

Create /etc/openvpn/server.conf with the following content. Each directive is commented.

port 1194
proto udp
dev tun

ca   /etc/openvpn/ca.crt
cert /etc/openvpn/server.crt
key  /etc/openvpn/server.key
dh   /etc/openvpn/dh.pem

tls-auth /etc/openvpn/ta.key 0
key-direction 0

server 10.8.0.0 255.255.255.0
ifconfig-pool-persist /var/log/openvpn/ipp.txt

# Push default route (full tunnel)
push "redirect-gateway def1 bypass-dhcp"
push "dhcp-option DNS 9.9.9.9"
push "dhcp-option DNS 149.112.112.112"

keepalive 10 120
cipher AES-256-GCM
tls-version-min 1.2

user nobody
group nogroup
persist-key
persist-tun

status  /var/log/openvpn/openvpn-status.log
log-append /var/log/openvpn/openvpn.log
verb 3

Enable and start the service:

mkdir -p /var/log/openvpn
systemctl enable --now openvpn@server
systemctl status openvpn@server

Full tunnel (redirect-gateway def1): all client traffic goes through the VPN. Useful for protecting users on untrusted networks or for unifying Internet exits. Downside: the VPS bandwidth becomes the bottleneck for all flows.

Split tunneling: only traffic destined for your private network (e.g. 10.8.0.0/24) goes through the VPN. The rest exits directly from the client. To enable it, replace the redirect-gateway line with specific routes in server.conf:

push "route 192.168.1.0 255.255.255.0"

This reduces load on the server and improves overall performance, but connections outside the VPN are no longer protected.

A standalone .ovpn file embeds all certificates. Generate it as follows:

cat > /tmp/client1.ovpn << EOF
client
dev tun
proto udp
remote <YOUR-VPS-IP> 1194
resolv-retry infinite
nobind
persist-key
persist-tun
remote-cert-tls server
cipher AES-256-GCM
verb 3
key-direction 1
<ca>
$(cat /etc/openvpn/easy-rsa/pki/ca.crt)
</ca>
<cert>
$(cat /etc/openvpn/easy-rsa/pki/issued/client1.crt)
</cert>
<key>
$(cat /etc/openvpn/easy-rsa/pki/private/client1.key)
</key>
<tls-auth>
$(cat /etc/openvpn/ta.key)
</tls-auth>
EOF

Transfer this file to the client via a secure channel (SCP or a single-use link). Then simply import it into OpenVPN Connect (Windows, macOS, Android, iOS) or NetworkManager (Linux) to establish the connection.

ufw firewall — open only what is necessary:

ufw allow ssh
ufw allow 1194/udp
ufw enable

If you use TCP 443 instead of UDP 1194, adapt the rule.

fail2ban for the OpenVPN port:

Create /etc/fail2ban/jail.d/openvpn.conf:

[openvpn]
enabled  = true
port     = 1194
protocol = udp
filter   = openvpn
logpath  = /var/log/openvpn/openvpn.log
maxretry = 5
bantime  = 3600

Then create the filter /etc/fail2ban/filter.d/openvpn.conf:

[Definition]
failregex = .*TLS Error: TLS key negotiation failed.*<HOST>.*
            .*VERIFY ERROR:.*<HOST>.*

Activate:

systemctl restart fail2ban
fail2ban-client status openvpn

Disable IPv6 if not needed: an unprotected IPv6 interface can bypass your tunnel. If your infrastructure does not need IPv6:

echo 'net.ipv6.conf.all.disable_ipv6=1' >> /etc/sysctl.conf
echo 'net.ipv6.conf.default.disable_ipv6=1' >> /etc/sysctl.conf
sysctl -p

MTU and fragmentation: OpenVPN encryption adds headers. By default the tunnel MTU (tun-mtu) is 1500, but UDP encapsulation can cause fragmentation on some networks. To avoid this, add to server.conf:

tun-mtu 1400
mssfix 1360

If you observe slow transfers or drops on connections with a low MTU (mobile networks, some corporate VPNs), reduce tun-mtu to 1300.

Cipher choice: AES-256-GCM is recommended for modern Linux kernels, which benefit from AES-NI hardware instructions (check with grep -m1 'aes' /proc/cpuinfo). For highly CPU-constrained servers, AES-128-GCM halves the encryption cost without significant security reduction for common use cases.

UDP or TCP: UDP 1194 is the default and most performant mode. TCP 443 is useful to bypass restrictive corporate firewalls, but introduces the "TCP over TCP" problem (redundant retransmissions) which degrades performance under packet loss.

TLS Error: TLS key negotiation failed

This error appears when the client cannot reach the server on the configured port. Check that the port is open (ss -ulnp | grep 1194), that the firewall rule is active, and that the ta.key key is identical on the server side (key-direction 0) and the client side (key-direction 1).

Routes not applied after connection

If traffic does not pass through the tunnel despite an established connection, check that IP forwarding is active (cat /proc/sys/net/ipv4/ip_forward must return 1) and that a NAT rule is in place:

iptables -t nat -A POSTROUTING -s 10.8.0.0/24 -o eth0 -j MASQUERADE

Replace eth0 with the main interface of your VPS (ip route get 8.8.8.8 to identify it).

Connections dropping regularly

If the tunnel disconnects every 2 minutes, check the keepalive in server.conf (keepalive 10 120: ping every 10 s, timeout at 120 s). On the client side, add ping-restart 60 to the .ovpn profile to force a clean reconnection before the server timeout.

VERIFY ERROR: depth=0, error=certificate is not yet valid

The server or client clock is out of sync. Synchronize NTP: timedatectl set-ntp true.