On October 28, I ran three SSH attempts against the original entry point of the fixed server localhost-1: two timed out. Switching to the LAN address 192.168.9.29, three consecutive attempts succeeded, in 0.8 to 0.9 seconds.
Same tailnet, same machine, one path works and the other doesn’t. The difference is that “direct connection” here means Tailscale’s NAT traversal path, and once traversal succeeds it goes quiet — stay quiet for a few minutes, send one more packet, and it has to punch through again, and wait again.
There’s a matching upstream report: tailscale/tailscale#20452, failures on idle paths. The direction is clear: if NAT hole punching has never been stable, stop treating it as the main path.
First, separate the three kinds of things
There are roughly three kinds of paths by which Tailscale connects two machines:
- Direct (hole punching succeeded, UDP peer-to-peer);
- Through Peer Relay, forwarded by a machine in the tailnet willing to act as a relay;
- To DERP, an official or self-hosted relay node.
Day to day, 1 is fast, so the problem only surfaces “at the moment 1 doesn’t hold.” Stacking more reliability onto 1 (keepalives, retries) doesn’t fix the root; what you can do is have 2 and 3 ready, so that fallback doesn’t depend on the luck of public UDP.
This round touched two machines:
localhost-3public103.11.77.24:40000/UDP, acting as Peer Relay;- spare VPS
38.207.161.228, running private DERP on 443/TCP, 80/TCP only for certificate validation, 3478/UDP for STUN.
For the admin console policy I chose the narrowest possible one: existing service and SSH rules untouched, only granting localhost-1 authorization to use these two relays. The full policy from before the change was backed up as tailnet-before.hujson, and at the end read back from the JSON editor and parsed against the final file — they have to match.
Who has to configure whom for Peer Relay
Peer Relay isn’t “install it and someone will use it.” The client picks it on its own, provided the admin console policy allows it, so the authorization has to hang on the machines that will use it, not on the relay. After adding the two relays to localhost-1's authorization, I confirmed the console had actually pushed out the region.
On the private DERP side I added a region 900. All official regions are kept; after the console pushed it out, the DERP map had 29 regions in total — additions only, no removals, and rollback is just deleting the region 900 entry.
The DERP node runs -verify-clients, with the local tailscaled verifying that clients belong to the current tailnet. The certificate is a Let’s Encrypt-issued IP certificate, with 38.207.161.228 in the SAN, managed and renewed by derper itself; the cache directory /var/lib/derper/certs/ can’t be copied straight into diagnostic logs.
Both WARP and tailscaled want the default route
This is the place in this round most likely to “break the thing you just fixed” again.
The WARP client modifies the default route, and tailscaled also picks its egress by its own routing table. When both sides think they own the default route, Tailscale traffic gets stuffed into the WARP tunnel and the relay becomes even less reachable.
The approach is to give the actual Tailscale routing table 52 a priority lookup: a policy route with priority 5208, plus a WARP firewall exception for tailscale0 only. Other WARP routes stay in effect, everything else untouched. Each machine has a 15-second timer guarding that exception, tailscale-warp-coexist.timer.
DERP needs to sign certificates, so ACME has to reach the public internet for validation. That egress goes through a dedicated table 65800 with priority 5207, refreshing only ACME’s current IPv4 every 15 minutes, leaving other default egress alone.
A checker that false-alarms every day
Once the maintenance script was up it started reporting “rule conflicts.” Digging in, it was a parsing problem: ip route prints dst and dstlen separately, and the script compared each field on its own, so the same subnet could naturally never match.
The fix was to normalize the subnet before comparing. After the fix I re-ran both repeated execution and timer triggering; both machines came back Result=success, ExecMainStatus=0, and the routes and firewall exceptions from the initial install were still there — not wiped out in passing by the “now fixed” script.
A watchdog that cries every day is worse than no watchdog: it trains people to ignore it.
Verification has to go down to the byte
The first time after configuring, all three SSH attempts connected: 9.00 s, 7.96 s, 9.35 s. Ten times slower than the LAN, but they connected.
That wasn’t the end of it, because “those few successes just now” don’t prove it will still connect after idling. So the verification was made actively adversarial:
- have the server side wake the connection with a reverse probe;
- transfer a full 262144 bytes through
103.11.77.24:40000's Peer Relay, SHA2568a39d2ab..., identical on both ends; - confirm in the relay session stats that there is real data in both directions.
Only then came the idle test: leave the connection sitting for over five minutes, no pings, no waking, and have the first request download 262144 bytes directly. Received in full, in about 28.84 seconds, checksum matching.
(This is part 1 of 2 of the same post.)
On the DERP side: public HTTP returns 204, HTTPS /derp/probe returns 200, client certificate verification result 0, issuer Let’s Encrypt YE1. On a Mac, tailscale debug derp 900 confirmed a connection could be established.
The warnings I couldn’t get around
The official derpprobe raises two warnings about the IP certificate: empty SNI, and a short-lived certificate valid for less than seven days.
Both are real. Rather than turning off certificate verification to make it go green, I noted that “forwarding effectiveness is judged by real connection tests.” Along the way I also tried force-prefer-derp 900 to force traffic through the private DERP; it does work, but it’s a debugging preference, not something to keep as a long-term setting, so exiting restores the default.
A note on versions too: all three machines run Tailscale 1.104.2, and on 10/28 the official release shipped 1.104.4 with a Tailscale SSH security fix. Built-in Tailscale SSH was already off on all three — they use OpenSSH — so no upgrade this round, but I noted that a future upgrade must rebuild derper at the matching version.
Boundaries
- The private DERP was verified for public TLS, client connections, and config delivery, but the full isolated switch to private-DERP-only was not done. The “peer-relay-upload-result.json” data is Peer Relay’s and can’t serve as evidence of DERP forwarding.
- One successful cold start doesn’t mean the idle failure is gone; daily use still needs watching.
- There’s no IPv6 on the machine, so the “no suitable address” hint during diagnosis doesn’t mean IPv4 is broken.
The rollback list is in the notes, four items: restore the backend policy tailnet-before.hujson, remove the proxy config with localhost-1, stop Peer Relay and close the UDP allowance with localhost-3, and on the standby VPS keep the binary but stop the service. The SSH public-key alias for the short hostname is removed with ssh-keygen -R localhost-1.
Wrapping up
After this round, the original entry point, the LAN entry point, the short hostname, and the full domain all log in, with strict host key checking left on the whole time — before adding the host key for localhost-1, I confirmed the DNS pointed at that machine, then added the same already-trusted public key as an alias.
There’s only one conclusion that’s actually useful: when hole punching is flaky, stop fixing hole punching. Build a fallback that works “even if it never connects,” then verify it over and over.
