Why self-host your remote desktop server
Commercial solutions like TeamViewer and AnyDesk share one characteristic: every connection passes through their relay infrastructure, regardless of the actual distance between the two machines. This creates three concrete problems.
Privacy and compliance. Data flows through third-party servers subject to the legislation of their host country. For IT providers accessing financial, medical or legal systems, this implicit delegation can invalidate a compliance agreement (GDPR, ISO 27001, HIPAA). Self-hosted RustDesk eliminates this dependency: encrypted traffic stays in your datacenter.
Vulnerability history. TeamViewer has experienced several notable incidents — in 2016, thousands of accounts were compromised via credential stuffing; in 2024, an intrusion into the corporate environment was attributed to an APT group. These incidents do not implicate the protocol, but highlight that depending on a third party means depending on their attack surface.
License cost. TeamViewer Business starts at over €600/year for a single concurrent user. AnyDesk is cheaper but still follows a per-seat model. Self-hosted RustDesk reduces costs to your VPS alone — a ServOrbit VPS starting at 99 DH/month handles an entire team.
What self-hosted RustDesk gives you
- No session limits — no per-seat license or concurrent connection quota; your server serves as many users as your VPS can handle
- Guaranteed end-to-end encryption — Ed25519 curve for key exchange, ChaCha20-Poly1305 for the video stream and keyboard/mouse inputs: your server's public key is embedded in the client, making interception structurally impossible
- Controlled latency — your relay is in the datacenter of your choice; by picking the region closest to your teams, RTT is consistently lower than a generic SaaS relay based in Northern Europe
- Cross-platform clients — Windows, macOS, Linux, Android, iOS and web browser from a single central server, without different configurations per platform
- Custom IDs — assign memorable identifiers (
SERVER-DB-01,LAPTOP-ALICE) instead of the random numeric IDs generated by default - Built-in REST API — RustDesk Server Pro exposes a REST API to query connection status, list registered machines and automate provisioning via your own scripts or a CMDB
- Full session audit log — every connection, disconnection and file transfer is timestamped and logged server-side; essential for environments with traceability requirements
RustDesk architecture: hbbs and hbbr
RustDesk separates the signaling role from the relay role into two distinct processes.
hbbs — the rendezvous server (ID Server). This is the entry point. Each RustDesk client that starts registers with a numeric (or custom) identifier. When a connection is initiated, both parties contact hbbs to exchange IP addresses and attempt a direct P2P connection. hbbs listens on TCP ports 21115 and 21116, as well as UDP 21116.
hbbr — the relay server. If the P2P connection fails (symmetric NAT, restrictive firewall), traffic is relayed via hbbr. The relay only sees encrypted streams — it cannot read the content. hbbr listens on TCP port 21117.
Connection flow. Machine A contacts hbbs via TCP to obtain machine B's address. Both machines attempt a direct UDP connection (hole punching). If it succeeds, hbbr is not involved and latency is minimal. If it fails, hbbs provides A with the address of hbbr to relay — with a slight overhead of a few milliseconds. In practice, P2P succeeds on the majority of home and enterprise networks that don't have strict NAT.
VPS requirements
Minimum resources. 1 GB RAM and 1 vCPU are sufficient for up to approximately 20 simultaneous relay connections. Beyond that, the limiting factor is bandwidth: a Full HD stream encodes at approximately 2-4 Mbit/s. For 50 simultaneous relay connections, plan for 2 vCPU, 2 GB RAM and a VPS with a guaranteed throughput of at least 200 Mbit/s.
Ports to open.
- TCP 21115 — NAT negotiation
- TCP 21116 — ID registration
- UDP 21116 — P2P hole punching
- TCP 21117 — relayed stream
- TCP 21118 and 21119 — web client (optional)
- TCP 443 — admin HTTPS interface (if enabled)
Required software. Docker Engine 24+ and Docker Compose v2 (docker compose without hyphen). A domain name or subdomain is recommended to expose the admin interface behind TLS — otherwise admin connections use plain HTTP.
Deploy RustDesk Server on a VPS
Create the directory and docker-compose.yml
Create the directory structure and Compose file:
mkdir -p /opt/rustdesk/data cd /opt/rustdeskContents of
docker-compose.yml:services: hbbs: image: rustdesk/rustdesk-server:latest command: hbbs ports: - "21115:21115" - "21116:21116" - "21116:21116/udp" - "21118:21118" volumes: - ./data:/root restart: unless-stopped depends_on: - hbbr hbbr: image: rustdesk/rustdesk-server:latest command: hbbr ports: - "21117:21117" - "21119:21119" volumes: - ./data:/root restart: unless-stoppedThe shared
./datavolume ensures thathbbsandhbbruse the same key pair generated on first startup.Start containers and retrieve keys
Start both services in the background:
docker compose up -dOn first startup,
hbbsautomatically generates an Ed25519 key pair in./data/. Retrieve the public key:cat /opt/rustdesk/data/id_ed25519.pubNote this value — you'll need it to configure each client. Without it, clients will refuse to connect to your server. Also keep
id_ed25519(private key) in a secure location; if it's compromised, all clients will need to be reconfigured with a new key.Configure the firewall (TCP/UDP ports)
Open the necessary ports in
ufw:ufw allow 21115/tcp ufw allow 21116/tcp ufw allow 21116/udp ufw allow 21117/tcp ufw allow 21118/tcp ufw allow 21119/tcp ufw reloadIf your VPS is also protected by a network firewall in the ServOrbit panel, apply the same rules there. Then verify that the ports are listening:
ss -tlnup | grep -E '2111[5-9]'Expose the admin interface behind an HTTPS reverse proxy
RustDesk Server Pro provides a web admin interface on port 21114 (HTTP). To expose it over HTTPS, add nginx as a reverse proxy.
Minimal nginx vhost for
rustdesk.your-domain.com:server { listen 443 ssl; server_name rustdesk.your-domain.com; ssl_certificate /etc/letsencrypt/live/rustdesk.your-domain.com/fullchain.pem; ssl_certificate_key /etc/letsencrypt/live/rustdesk.your-domain.com/privkey.pem; location / { proxy_pass http://127.0.0.1:21114; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; } }Obtain the certificate with Certbot:
certbot --nginx -d rustdesk.your-domain.comConfigure desktop clients (Windows, macOS, Linux)
Download the RustDesk client from [rustdesk.com](https://rustdesk.com) and open the network settings (gear icon → Network). Enter:
- ID Server: your VPS IP address or domain name
- Relay Server: same value
- Key: the value copied fromid_ed25519.pubConfirm. The connection indicator turns green if the server is reachable and the key is correct. From this point, no connection attempts will go through the public RustDesk servers.
Configure mobile clients (Android and iOS)
On Android, install RustDesk from the Play Store or directly via APK (useful if the Play Store is restricted on some enterprise devices). On iOS, RustDesk is available on the App Store.
In both cases, go to Settings → ID/Relay Server and enter the same values as for the desktop client: server address and public key. The key can be shared via QR code from the RustDesk Server Pro admin interface, which simplifies mass deployment on mobile fleets.
Test an end-to-end connection
On two machines configured with your server, note the identifier displayed on the target machine. From the source machine, enter this identifier in the RustDesk connection field and initiate the session.
Check the logs to confirm the connection is going through your infrastructure:
docker logs rustdesk-hbbr-1 --tail 20A line containing
[RELAY]indicates the relay is being used (strict NAT). A[DIRECT]line or no line in thehbbrlogs indicates a direct P2P connection — ideal for latency.
Configuring TLS for secure connections
By default, RustDesk streams are end-to-end encrypted with ChaCha20-Poly1305 — additional TLS is not required for the remote desktop sessions themselves. However, the admin interface (port 21114) and the REST API run in plain HTTP if you don't protect them.
Option 1: Certbot + nginx (recommended). As shown in step 4, nginx acts as a reverse proxy in front of port 21114. Certbot renews the certificate automatically via a systemd timer.
Option 2: Traefik as edge proxy. If you already use Traefik for other services on the same VPS, simply add labels to the hbbr service in Compose:
labels:
- "traefik.enable=true"
- "traefik.http.routers.rustdesk-admin.rule=Host(`rustdesk.your-domain.com`)"
- "traefik.http.routers.rustdesk-admin.tls.certresolver=letsencrypt"
- "traefik.http.services.rustdesk-admin.loadbalancer.server.port=21114"What stays unencrypted by design. Ports 21115-21119 don't use HTTPS because they use their own binary protocols. Encryption of these streams is handled by Ed25519 and ChaCha20 keys — not TLS. Do not attempt to force HTTPS on these ports.
Enabling authentication and access control
The open source version of RustDesk Server has no native authentication interface — any client that knows your public key can attempt to connect to a registered machine (if that machine accepts the request). RustDesk Server Pro (self-hostable, free for up to 20 users) adds several control layers.
User accounts and groups. The admin interface lets you create named accounts, group them and define access policies per group. A technician can have access only to the 'Infrastructure' group machines, not HR workstations.
Connection approval. Each machine can be configured to require physical confirmation from the local user before allowing remote control — the 'Accept connection' dialog appears on the target machine's screen.
Full audit trail. Every event (connection initiated, accepted, refused, duration, file transfer) is recorded with the initiator's identity and timestamp. Logs are exportable as CSV from the admin interface.
Disabling public servers. To force exclusive use of your server — without falling back to public RustDesk relays if your server is unavailable — add the --key option with your private key to the hbbs command. Clients configured with your public key will then refuse any alternative server, including the official RustDesk server.
Optimizing image quality and performance
RustDesk uses its own video codec by default (VP9 or H.264 depending on the platform) with adaptive quality based on detected bandwidth. Several levers let you tune this behavior.
Image quality. In the client settings, 'Image quality' can be set to Low, Medium, High or Custom. In enterprise environments on a LAN or a nearby VPS, 'High' is usable without saturating bandwidth. On a mobile or long-distance connection, 'Medium' reduces perceived latency.
Codec. RustDesk automatically selects the most suitable codec: H.265 if both endpoints have hardware encoding/decoding, VP9 otherwise. On a VPS without a GPU, VP9 is the default — reasonable in CPU for office-type streams. Avoid forcing H.264 on a low-CPU VPS if you have multiple simultaneous connections.
P2P vs relay. A direct P2P connection is always preferable: it avoids the extra hop through hbbr, removing the relay intermediary and lowering round-trip time. If your users connect from the same LAN as the target machine, check that the firewall doesn't block direct UDP connections between hosts — RustDesk tries P2P first before falling back to relay.
RustDesk vs TeamViewer vs AnyDesk vs Guacamole
Scroll the table
| Criteria | RustDesk (self-hosted) | TeamViewer | AnyDesk | Guacamole |
|---|---|---|---|---|
| Hosting model | Self-hosted (VPS) | SaaS (cloud vendor) | SaaS (cloud vendor) | Self-hosted (web server) |
| License cost | Free (OSS) / Pro paid | From €600/year/user | From €180/year/user | Free (Apache 2.0) |
| Protocol | Proprietary encrypted (Ed25519 + ChaCha20) | Proprietary (RDP/VNC hybrid) | DeskRT (proprietary codec) | VNC / RDP / SSH via browser |
| P2P connection | Yes (with relay fallback) | No (always via cloud) | Yes (with fallback) | No (central proxy) |
| Native mobile clients | Android + iOS native | Android + iOS native | Android + iOS native | Browser only |
| Audit log | Yes (Pro) | Yes (Business+) | Yes (Business+) | Limited (raw server logs) |
| REST API | Yes (Pro) | Yes (paid) | No (third-party API) | Yes (partial REST) |
| Data sovereignty | Total (your infra) | None (TeamViewer cloud) | Partial (AnyDesk relay) | Total (your infra) |
Enterprise deployment: mass-provisioning clients
RustDesk supports automatic configuration via a configuration file embedded in the client executable. When building or packaging for your fleet, place a RustDesk2.toml file in the configuration directory with your values:
relay-server = "your-vps.example.com"
id-server = "your-vps.example.com"
key = "<your_ed25519_public_key>"On Windows, this file goes in %APPDATA%\RustDesk\config\. On Linux, in ~/.config/rustdesk/. You can deploy this configuration via your fleet management tool (Ansible, Intune, Puppet) without user interaction. The public key and server address are pre-loaded — the user installs RustDesk and is immediately connected to your infrastructure.
Monitoring and maintaining the server
Docker metrics. Monitor resource consumption in real time with docker stats — a CPU spike on hbbr indicates an unusually high number of relayed connections. If the average exceeds 80% for several minutes, consider increasing vCPU or checking whether stuck connections are generating retry loops.
Log rotation. Containers write to stdout/stderr, managed by the Docker daemon. Add a rotation policy in docker-compose.yml to prevent logs from filling the disk:
logging:
driver: "json-file"
options:
max-size: "10m"
max-file: "5"Image updates. RustDesk publishes new versions regularly. To update with minimal perceived downtime:
docker compose pull
docker compose up -dDocker replaces containers one by one. Sessions in progress via hbbr are dropped during the restart (a few seconds) — prefer updates during off-peak hours. Keys in ./data/ are preserved — no client reconfiguration required.
Troubleshooting common issues
Connection impossible, ID not found. Check that hbbs is running (docker ps) and that TCP ports 21115-21116 and UDP 21116 are reachable from outside: nc -vz <vps-ip> 21116. If the test fails, the network firewall (ufw or VPS rule) is blocking the port.
'Incorrect key' on the client side. The key shown in the client doesn't match id_ed25519.pub on the server. Re-read the exact value with cat /opt/rustdesk/data/id_ed25519.pub and reconfigure the client — watch for stray spaces or newlines during copy-paste.
Always in relay mode, never P2P. UDP hole punching fails if one of the two networks uses symmetric NAT (common on some 4G/5G mobile networks and corporate VPNs). This is not a bug — the relay works correctly, with slightly higher latency. To force a diagnosis, compare hbbr logs with and without VPN active on the client side.
Admin interface unreachable. Port 21114 is not exposed in the base docker-compose.yml — it must be added explicitly or accessed through the nginx reverse proxy shown in step 4. Also check that you're using the rustdesk-server-pro image (the open source version has no admin interface).
Image update without regenerating keys. After docker compose pull && docker compose up -d, verify that ./data/id_ed25519.pub hasn't changed — it shouldn't, but volume corruption can in rare cases regenerate the pair. If the key has changed, all clients will need to be reconfigured.
