Tailnet Zero-Downtime Deployments
-
Tailscale is amazing for enabling private mesh network overlays. For connecting your phone to a server you having running at home, easily and securely. If you’re building things you want to deploy on your tailnet, what’s the best way to do that?
I want everything to run on my own hardware, minimizing my costs and letting me use what I already have. Tailscale lets me talk with my computers easily. Kamal lets me deploy from my local machine with ease and zero downtime (my wife will never know!). Tailscale’s Serve feature work great with Kamal Proxy’s accessory setup.
Kamal has one onerous requirement, it requires a container registry. This registry has to be available from your laptop and server. If you use an external service, like GitHub’s Artifact Repository or Digital Ocean’s container registry, then you should be good to go. If you, like me, want to also run your own container registry on your tailnet, I wrote another article about that.
How it works
Using a ephermeral tailscale auth key, Tailscale will route traffic on our network to the accessory tailscale container that Kamal deploys for us. We use docker’s dns routing to point Tailscale’s Serve feature to the kamal-proxy container that Kamal deploys. Kamal’s proxy will check the hostname for us then route to the application container for our app. Because we’re deploying behind kamal-proxy, Kamal will handle zero downtime deploys for us, no configuration needed!
Limitations
Kamal proxy routes via the hostname for you, so you can’t access these services via the IP address.
Deploy more services
Kamal Proxy can handle multiple apps deployed to the same server. We register a different tailscale sidecar for each one to make our deployments simpler and each service can have it’s own identifier in our Tailscale Admin Dashboard. Requests to both services will run through the same kamal proxy container, who will route it properly via hostname.
Appendix A: Configuration
# excerpts from my deploy.yml for Kamal
# my server is on tailscale as well
servers:
web:
- <server name>.<tailnet name>.ts.net
# To connect with Zot on my tailnet
build:
driver: docker
# tell the proxy my tailscale hostname
proxy:
# tailscale terminates the ssl connection
ssl: false
# same hostname as the tailscale sidecar
host: <service name>.<tailnet name>.ts.net
# rails port number
app_port: 3000
# rails takes a long time to boot on my little server...
healthcheck:
path: /up
interval: 5
timeout: 600
run:
# don't publish ports to the host machine,
# all the traffic will come through the docker network
# via the tailscale accessory
publish: false
accessories:
sidecar:
image: ghcr.io/tailscale/tailscale:stable
# for kamal, the host to run this sidecar on, same as servers.web configuration
host: <server name>.<tailnet name>.ts.net
files:
# I don't think directories works the way I way, so
# copy and mount the file directly.
- config/tailscale/ts.json:/config/ts.json
env:
clear:
TS_USERSPACE: false
TS_STATE_DIR: /var/lib/tailscale
TS_SERVE_CONFIG: /config/ts.json
TS_EXTRA_ARGS: --advertise-tags=tag:container
secret:
# loaded with .kamal/secrets
- TS_AUTHKEY
options:
# must be the same as the proxy's service name so tailscale and kamal proxy agree
hostname: <service name>
cap-add: NET_ADMIN
device: /dev/net/tun:/dev/net/tun
# derive the volume from the service name so we can have multiple tailscale sidecars on the same server
volume: <service name>-tailscale-state:/var/lib/tailscale