RustDesk self hosted setup — full Docker + Caddy TLS walkthrough

You're trying to self-host RustDesk but keep hitting the same blockers: NAT and firewalls, confusing server components, and the nagging question of whether you need TLS on top of RustDesk's own encryption. This guide walks a technically minded reader through a complete, reproducible rustdesk self hosted setup on a single VPS — Docker Compose for hbbs/hbbr, and TLS fronting using Caddy so clients can connect on 443 from restrictive networks.

What we're building and why

The goal: a single-VPS RustDesk rendezvous + relay that your clients point at by DNS name. Components in play:

• rustdesk server components (hbbs and hbbr) running in Docker — tested here with rustdesk-server v1.2.1 images.
• Caddy v2 (certificate management via Let's Encrypt) to provide a TLS front door on port 443 so clients can reach your relay even when outbound 21115 is blocked.
• Optional UDP passthrough or additional relays for scale (covered in notes).

Why do this? RustDesk’s protocol already provides end-to-end crypto for the desktop session, but many networks only allow outbound 443/80. Terminating TLS at your VPS (and letting Caddy obtain and auto-renew certs) is a practical way to make the service reachable without opening nonstandard ports. If you just want LAN access or control NAT with reverse tunnels, see our article on remote-desktop-without-port-forwarding.

Prerequisites and choices

What I used while writing this:

• Ubuntu 22.04 LTS on a public VPS (1 vCPU / 2 GB RAM is enough for test use; for many clients scale up to 4+ CPU and monitor CPU/RAM).
• Docker 24.x and Docker Compose v2 (compose V2 CLI syntax).
• rustdesk-server Docker image (tag v1.2.1 in this guide) — adjust if newer stable releases exist.
• Caddy v2.6+ (Caddy's automatic ACME makes cert renewals painless).
• A DNS A record like rustdesk.example.com pointing at your VPS public IP, and ports 80/443 allowed (Caddy needs 80/443 to validate).

Notes on when a hosted commercial product is a better fit: if you need enterprise-grade SLA, advanced session auditing, or commercial support, TeamViewer/AnyDesk may be better — see rustdesk-vs-anydesk and our price comparison pieces. Self-hosting is best when you want control, lower recurring costs, or to avoid third-party servers.

Step 1 — Deploy rustdesk server with Docker Compose

Create a project directory on the VPS and drop in the docker-compose.yml below. This example exposes the typical RustDesk server ports to the host. Replace environment variables and image tags to suit your environment.

mkdir -p ~/rustdesk-server
cd ~/rustdesk-server
cat > docker-compose.yml <<'EOF'
version: '3.8'
services:
  rustdesk-server:
    image: rustdesk/rustdesk-server:1.2.1
    container_name: rustdesk-server
    restart: unless-stopped
    ports:
      - "21115:21115/tcp"   # rendezvous / TCP relay
      - "21115:21115/udp"   # optional UDP relay (if your image supports it)
      - "21116:21116/udp"   # additional UDP (some builds use multiple UDP ports)
    volumes:
      - ./data:/root/.config/rustdesk-server
    environment:
      - RUSTDESK_RELAY_IPV4=0.0.0.0
      - RUSTDESK_RELAY_PORT=21115
EOF

Bring it up:

docker compose up -d
# watch logs
docker compose logs -f rustdesk-server

Confirm the container started and is listening on 21115. On your VPS run:

ss -tuln | grep 21115

If the image you use exposes different ports or has a different configuration style, consult that image's README. Some operators compile hbbs/hbbr manually and use custom ports; adapt these steps accordingly.

Step 2 — Use Caddy to put TLS on top of the relay (443)

There are two common patterns to make RustDesk reachable on 443:

1. TCP TLS termination with Caddy 2.6+ (Caddy terminates TLS and forwards raw TCP to the rustdesk relay). Caddy added improved TCP features in v2.6; if you use that approach, you get automatic ACME certificates and renewals.
2. Let Caddy manage certificates only, and use a TCP proxy (stunnel, HAProxy, or Nginx stream) to consume those cert files. This is a fallback if your Caddy version or environment doesn't support the TCP proxy features you need.

Below is a minimal, pragmatic Caddy setup that uses Caddy to do TLS on port 443 and forward the raw TCP connection to the RustDesk relay at localhost:21115. This requires Caddy v2.6+ (check with caddy version).

# Caddyfile (save as ./Caddyfile in the same folder as docker-compose.yml)
rustdesk.example.com:443 {
    # Caddy will obtain/renew certificates automatically
    reverse_proxy 127.0.0.1:21115 {
        # Use plain TCP transport; Caddy will accept TLS from clients and connect to the backend via TCP
        transport http {
            # We don't speak HTTP to the backend; keep minimal transport settings
        }
    }
}

Docker-compose snippet to run Caddy next to the rustdesk server (same project):

cat >> docker-compose.yml <<'EOF'
  caddy:
    image: caddy:2.6.4
    container_name: caddy
    restart: unless-stopped
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - ./Caddyfile:/etc/caddy/Caddyfile:ro
      - caddy_data:/data
      - caddy_config:/config
volumes:
  caddy_data:
  caddy_config:
EOF

docker compose up -d caddy

Important caveat: Caddy's built-in reverse_proxy is HTTP-oriented, so behavior when proxying non-HTTP raw TCP depends on the Caddy version and the transport options. In practice many operators use Caddy's TCP proxying features introduced in 2.6 to terminate TLS for arbitrary TCP backends. If you run into protocol issues, use option (2) above: let Caddy handle certs and hand the cert files to a small TCP TLS layer (stunnel or HAProxy stream) which then proxies plain TCP to rustdesk.

Example: Caddy as cert manager + stunnel for TLS termination (quick overview).

# 1) Ensure Caddy is running and has issued certs for rustdesk.example.com
# 2) Copy cert/key from Caddy storage into a place stunnel can read, or mount the same volume.
# 3) Run stunnel with a config that points TLS 443 -> 127.0.0.1:21115

# stunnel.conf snippet
[rustdesk]
accept = 443
connect = 127.0.0.1:21115
cert = /etc/stunnel/fullchain.pem
key = /etc/stunnel/privkey.pem

# In Docker world, mount Caddy's certificate files into the stunnel container path above.

Either approach gives you a hostname (rustdesk.example.com) and port 443 for clients. Test connectivity from another machine with: nc -vz rustdesk.example.com 443 — you should establish a TLS handshake if Caddy/stunnel are configured correctly.

Step 3 — Point clients at your self-hosted server

RustDesk clients allow custom ID/relay servers in settings. The exact UI varies by platform and version, but the essential values are:

• ID Server / Rendezvous: rustdesk.example.com:443 (or your domain and port)
• Relay Server: rustdesk.example.com:443

On Windows: Open RustDesk > Settings > ID/Server and set Id server and Relay server to rustdesk.example.com:443. On Linux and macOS the same settings exist in Preferences. For headless installs, the client can sometimes be supplied with command-line flags or a config file — check the client repo for details.

Be sure clients are recent (1.2.x or newer at time of writing). Newer clients include protocol fixes and better NAT traversal. If you attempt to use a much older client, behavior may vary.

Troubleshooting and hardening

Common issues and how to debug:

• Firewall: confirm VPS firewall (ufw/iptables/cloud provider) allows 80/443 inbound. For the RustDesk server container listening on 21115 locally, ensure the TCP socket exists (ss/netstat).
• Certificate issuance: if Let's Encrypt can't validate, Caddy will log an error. Confirm your DNS A record points to the VPS and that port 80 is reachable during initial issuance.
• Protocol mismatch: if you see TLS handshake success but connection failures from the client, you may be proxying HTTP-only with Caddy by mistake. Use the stunnel approach in those cases.
• UDP relay: RustDesk performance improves with UDP for screen frames; if UDP is needed and your network path blocks UDP, you’ll fall back to TCP relay. Expose/forward the UDP ports only if you control your network and know what you’re doing.

Security hardening tips:

• Run the rustdesk server under a dedicated unprivileged user and keep Docker images up to date.
• Enable fail2ban or similar to throttle repeated failed connections, and monitor logs (docker compose logs -f rustdesk-server).
• Back up your rustdesk server data directory (in the example it's ./data) regularly.
• Consider running the relay behind a private network or VPC if you are operating multiple relays.

Scaling notes: A single relay is fine for small teams. For larger deployments, run multiple hbbr processes on separate machines and use DNS load-balancing or a proper L4 load balancer. If you need centralized enterprise features like auditing, that’s when commercial solutions may have an edge. See our pieces on self-hosted-remote-desktop and rustdesk-vs-anydesk for tradeoffs.

Maintenance, costs, and final recommendations

Operational costs are mostly VPS fees and your time. A typical small VPS (1–2 CPU, 2–4 GB) can be found for $5–10/month (DigitalOcean, Vultr, Hetzner). Caddy and RustDesk server are open-source; the primary recurring cost is hosting. If you need GUI-based billing or enterprise support, commercial vendors charge per-seat or per-session — see our pricing comparisons such as anydesk-pricing-explained and godeskflow-vs-teamviewer-pricing for context.

Recommendations:

• Start with a single VPS and the Docker Compose layout above. Test connections from the exact networks your users will be on (home ISPs, corporate firewalls).
• If clients are frequently behind very restrictive firewalls, put your relay on 443 and verify TLS termination works reliably; use Caddy+stunnel if Caddy-only TCP proxying causes issues.
• Automate backups of the server data dir and track image upgrades. RustDesk server releases move; test upgrades in a staging environment if you rely on production uptime.

If you want a shorter walk-through for remote access patterns or alternatives, read our related articles: remote-access-setup-guide and remote-desktop-security.

Done reading? If you want to try a fully open-source remote desktop that integrates well with self-hosting, download GoDesk or compare its pricing — see GoDesk download and GoDesk pricing. If you’re ready to deploy this RustDesk setup, start by creating the docker-compose.yml and Caddyfile above and run docker compose up -d. For a quick start, go to /download to grab clients and get connected.