低價 VPS 的 80/443 連接埠受 NAT 封鎖，Cloudflare Tunnel 架構讓網站上線

1. The Symptom: When "Correct" Means Down

The browser gave me the dreaded ERR_CONNECTION_TIMED_OUT, alternating with intermittent Connection reset by peer.

My A record resolved cleanly to the IP listed in the hosting control panel. Inside the box, systemctl status caddy showed green across the board. The local firewall had ports open:

ufw status
# Status: active
# To                         Action      From
# --                         ------      ----
# 80/tcp                     ALLOW       Anywhere
# 443/tcp                    ALLOW       Anywhere

Yet hitting the site over the internet yielded dead silence. No handshakes, no access logs, no errors in Caddy’s journal. Packets were disappearing into the void before they ever reached the virtual network interface.

2. The Dead Ends

Before finding the real culprit, I burned an hour chasing ghosts:

1. Flushing DNS caches: I assumed propagation lag. I queried Cloudflare (1.1.1.1) and Google (8.8.8.8) directly via dig +short mydomain.com. The IP matched instantly.
2. Scouring Caddyfile syntax: I stripped the configuration down to a static respond "hello world" block, suspecting an edge-case redirect loop. Still unreachable.
3. Questioning Tailscale: I had installed Tailscale earlier for remote access. I wondered if MagicDNS or subnet routing had hijacked default routes or confused bind interfaces. Disabling Tailscale changed nothing.
4. Scrutinizing iptables: I dumped the raw packet chains (iptables -L -n -v). Packets on the host loopback incremented; WAN counters sat dead at zero.

3. The Diagnosis: The Two Commands That Cracked It

The realization clicked when I stepped back from application configs and checked basic network interfaces.

Check 1: Interface Inspection

ip addr show

The interface (eth0) didn't carry the public IP shown on my invoice. Instead, it held an address in the carrier-grade/shared range: 10.x.x.x (or 100.64.x.x).

The provider was using 1:N NAT. The public IP assigned to my account belonged to a perimeter gateway shared across multiple tenants. Unless the provider explicitly routed inbound traffic on low ports to my private IP, my machine would never receive an unsolicited TCP handshake on port 80 or 443.

Check 2: Loopback TLS Validation

To verify whether the server stack itself was healthy, I forced a local request against the local interface while passing the domain header:

curl -Iv https://localhost --resolve mydomain.com:443:127.0.0.1

The TLS handshake succeeded immediately. Caddy served the certificate, completed the negotiation, and returned the payload without complaint.

The verdict: The application was fine. Inbound routing was non-existent.

4. Why the Standard Fixes Fell Short

In this scenario, traditional workarounds fall into two buckets:

• Begging support for port forwarding: Some budget hosts offer NAT port mapping via their control panel. But they usually hand you non-standard ports (e.g., mapping external port 18443 to internal 443). That breaks zero-configuration web browsing; nobody wants to type mydomain.com:18443 into a mobile browser.
• Paying for a dedicated public IPv4: That often costs $2 to $4 a month extra. On a $3/month hobby box, that doubles or triples your recurring spend. It defeats the entire point of hunting down a bargain VPS.

Rather than pay more money or settle for ugly port numbers, I flipped the traffic model upside down: stop listening for incoming connections, and punch out instead.

5. The Architecture: Split-Horizon with Tunnels

The design splits traffic strictly between public visitors and private management:

Format:

[Public Web Visitor]
        │
        ▼ (HTTPS)
 [Cloudflare Edge]
        │
        ▼ (Encrypted QUIC / Outbound Tunnel)
[cloudflared daemon] ─── (localhost:443) ───► [Caddy Web Server] ───► [App]
                                                     ▲
                                                     │ (Tailscale IP only)
[My Laptop / Workstation] ── (Encrypted Mesh) ───────┴─── [Admin / SSH / DB]

Layer 1: Public Ingress (Cloudflare Tunnel + Caddy)

Instead of waiting for incoming SYN packets on ports 80/443, the lightweight cloudflared daemon runs locally on the VPS and establishes persistent, outbound-only connections to Cloudflare's edge data centers.

Because the connection originates from inside the box, the provider's NAT gateway allows it out freely. Public users hit Cloudflare, which shuttles requests down the tunnel to localhost.

To keep HTTPS clean between cloudflared and Caddy, Caddy acquires valid certificates using the DNS-01 ACME challenge (via the Cloudflare API plugin). This skips HTTP-01 port-80 challenges entirely:

# /etc/caddy/Caddyfile
mydomain.com {
    tls {
        dns cloudflare {env.CLOUDFLARE_API_TOKEN}
    }

    reverse_proxy localhost:3000
}

Layer 2: Private Control Plane (Tailscale)

SSH and administrative dashboards are stripped off the public internet entirely.

• The public firewall drops all inbound traffic on eth0.
• Tailscale binds to its virtual tailscale0 interface.
• SSH listens exclusively on the Tailscale mesh IP (100.x.y.z), secured by WireGuard keys and two-factor SSO.

No open ports. No brute-force bot sweeps hitting port 22.

6. The Gotchas That Cost Me the Most Time

Making the components talk to each other was fast. Dealing with subtle edge-case behaviors took hours.

Gotcha 1: Cloudflare 502 via Origin Server Name (SNI)

Once the tunnel came up, requests through the browser threw immediate 502 Bad Gateway errors.

If your tunnel points to https://localhost:443, cloudflared attempts a TLS handshake using localhost as the SNI. Caddy only holds certificates for mydomain.com, so it terminates the handshake.

The fix: In the Cloudflare Tunnel ingress configuration, explicitly configure the originServerName:

# ~/.cloudflared/config.yml
ingress:
  - hostname: mydomain.com
    service: https://localhost:443
    originRequest:
      originServerName: mydomain.com
  - service: http_status:404

Gotcha 2: Bot Fight Mode Killed Google's Sitemap Fetch

A few days later, Google Search Console threw recurring errors: "Couldn't fetch sitemap."

Cloudflare's default "Bot Fight Mode" aggressively challenges requests that fail standard browser heuristic checks. Google's crawler received managed challenge pages instead of XML. Turning off basic Bot Fight Mode in favor of granular WAF rules resolved crawler access immediately.

Gotcha 3: The Caching SPA Fallback Trap

My frontend router was configured to return index.html on 404s. During an asset deployment, a hashed CSS chunk (chunk-7a8b.css) failed to upload.

Cloudflare dutifully cached the 200 OK response containing index.html under the CSS file's URL. When the new bundle loaded, browsers attempted to parse HTML as CSS:

Refused to apply style from '...' because its MIME type ('text/html') is not a supported stylesheet MIME type.

I had to purge the Cloudflare cache and set strict cache-control headers in Caddy for static assets to keep fallback paths from being stored upstream.

7. Verification: Smoke Tests & Health Checks

To make sure everything stays intact across reboots or upstream outages, I keep a quick verification script (scripts/verify-networking.sh):

#!/usr/bin/env bash
set -euo pipefail

echo "==> 1. Checking local services..."
systemctl is-active --quiet caddy && echo "✔ Caddy is running" || echo "✘ Caddy down"
systemctl is-active --quiet cloudflared && echo "✔ cloudflared is running" || echo "✘ cloudflared down"
systemctl is-active --quiet tailscaled && echo "✔ tailscale is running" || echo "✘ tailscale down"

echo -e "\n==> 2. Testing local TLS resolution..."
HTTP_STATUS=$(curl -s -o /dev/null -w "%{http_code}" https://localhost \
  --resolve mydomain.com:443:127.0.0.1)
[[ "$HTTP_STATUS" =~ ^(200|301|302)$ ]] && echo "✔ Local TLS handshake OK ($HTTP_STATUS)" || echo "✘ Local TLS failed ($HTTP_STATUS)"

echo -e "\n==> 3. Testing public ingress..."
EDGE_STATUS=$(curl -s -o /dev/null -w "%{http_code}" https://mydomain.com)
[[ "$EDGE_STATUS" =~ ^(200|301|302)$ ]] && echo "✔ Public edge return OK ($EDGE_STATUS)" || echo "✘ Public edge failed ($EDGE_STATUS)"

echo -e "\n==> 4. Checking Tailscale connectivity..."
tailscale ping -c 1 my-laptop > /dev/null && echo "✔ Tailscale mesh reachable" || echo "✘ Tailscale unreachable"

Running this after kernel updates or container restarts takes five seconds and catches detached tunnels instantly.

8. Making It Repeatable: Packaging It as a Skill

After setting this up by hand, I didn't want to repeat the manual troubleshooting on my next low-cost VPS instance.

I codified the process into an agent skill definition (SKILL.md) that documents the exact checks, common configuration blocks, and diagnostic steps for automated runners and AI tooling. It acts as an executable runbook covering:

• Initial interface NAT sanity checks
• Automated Caddy installation with Cloudflare DNS plugins
• Systemd service templates for cloudflared
• Fail-safe firewall rule generation

The repository and automation manifests are available here:

👉 https://github.com/AnandaAp/vps-hybrid-networking

9. When This Fits (And When It's Overkill)

This pattern solved my problem cleanly, but it isn't a universal hammer.

This architecture shines when:

• You're using NAT VPS providers: If you're running on ultra-cheap VPS hosts, internal home servers, or shared-IP VPS tiers where port 80/443 ingress is blocked or charged at a premium.
• You want total port closure: You want zero exposed listening ports on the public interface, minimizing bot scanning and automated exploit probes.
• You need split administrative access: Tailscale provides clean, encrypted access to admin panels, metrics, and SSH without maintaining a public bastion host.

Skip it if:

• You run high-throughput raw TCP/UDP services: Cloudflare Free tunnels have bandwidth considerations and are designed primarily for HTTP/HTTPS/WebSockets. High-bandwidth media streaming or non-HTTP traffic (like game servers) will struggle here.
• Vendor lock-in is a dealbreaker: Relying on Cloudflare for ingress ties your DNS and perimeter routing directly to their ecosystem.
• You already have a routable static public IPv4: If inbound 80 and 443 are unobstructed, native Caddy with automated HTTP-01 challenges remains the simplest, most direct deployment path available.