Tailnet Zero-Downtime Deployments

-ben

Tailscale is amazing for enabling private mesh network overlays. For connecting your phone to a server you having running at home, easily and securely. If you’re building things you want to deploy on your tailnet, what’s the best way to do that?

I want everything to run on my own hardware, minimizing my costs and letting me use what I already have. Tailscale lets me talk with my computers easily. Kamal lets me deploy from my local machine with ease and zero downtime (my wife will never know!). Tailscale’s Serve feature work great with Kamal Proxy’s accessory setup.

Kamal has one onerous requirement, it requires a container registry. This registry has to be available from your laptop and server. If you use an external service, like GitHub’s Artifact Repository or Digital Ocean’s container registry, then you should be good to go. If you, like me, want to also run your own container registry on your tailnet, I wrote another article about that.

How it works

Using a ephermeral tailscale auth key, Tailscale will route traffic on our network to the accessory tailscale container that Kamal deploys for us. We use docker’s dns routing to point Tailscale’s Serve feature to the kamal-proxy container that Kamal deploys. Kamal’s proxy will check the hostname for us then route to the application container for our app. Because we’re deploying behind kamal-proxy, Kamal will handle zero downtime deploys for us, no configuration needed!

Limitations

Kamal proxy routes via the hostname for you, so you can’t access these services via the IP address.

Deploy more services

Kamal Proxy can handle multiple apps deployed to the same server. We register a different tailscale sidecar for each one to make our deployments simpler and each service can have it’s own identifier in our Tailscale Admin Dashboard. Requests to both services will run through the same kamal proxy container, who will route it properly via hostname.

Appendix A: Configuration

# excerpts from my deploy.yml for Kamal

# my server is on tailscale as well
servers:
  web:
    - <server name>.<tailnet name>.ts.net

# To connect with Zot on my tailnet
build:
  driver: docker

# tell the proxy my tailscale hostname
proxy:
  # tailscale terminates the ssl connection
  ssl: false
  # same hostname as the tailscale sidecar
  host: <service name>.<tailnet name>.ts.net
  # rails port number
  app_port: 3000
  # rails takes a long time to boot on my little server...
  healthcheck:
    path: /up
    interval: 5
    timeout: 600
  run:
    # don't publish ports to the host machine,
    # all the traffic will come through the docker network
    # via the tailscale accessory
    publish: false

accessories:
  sidecar:
    image: ghcr.io/tailscale/tailscale:stable
    # for kamal, the host to run this sidecar on, same as servers.web configuration
    host: <server name>.<tailnet name>.ts.net
    files:
      # I don't think directories works the way I way, so
      # copy and mount the file directly.
      - config/tailscale/ts.json:/config/ts.json
    env:
      clear:
        TS_USERSPACE: false
        TS_STATE_DIR: /var/lib/tailscale
        TS_SERVE_CONFIG: /config/ts.json
        TS_EXTRA_ARGS: --advertise-tags=tag:container
      secret:
        # loaded with .kamal/secrets
        - TS_AUTHKEY
    options:
      # must be the same as the proxy's service name so tailscale and kamal proxy agree
      hostname: <service name>
      cap-add: NET_ADMIN
      device: /dev/net/tun:/dev/net/tun
      # derive the volume from the service name so we can have multiple tailscale sidecars on the same server
      volume: <service name>-tailscale-state:/var/lib/tailscale