Tyblog | Too Simple To Fail: Marrying Nomad, Caddy, and Wireguard

1

My little lab can afford some experimental allowances given that I’ll never (hopefully) breach the “thousands of hosts” mark. One experiment that paid off recently was ditching Traefik v1 for a hybrid setup that uses Nomad, consul-template, Caddy, and wireguard in order to provide the HTTP routing layer for my services.

This is a companion discussion topic for the original entry at https://blog.tjll.net/too-simple-to-fail-nomad-caddy-wireguard/
2

Thank you for this write-up. I have nomad/consul/vault running “distributed system” on my 3 raspberry PI with SeaweedFS.
Have you looked Fabio? It seems to be specifically designed instead of caddy in such setup - I was excited about caddy, but then I had a bad experience when tried to deploy it on somebodies else jetson nano remotely.

4

I looked at Fabio briefly, but I was very keen on having my reverse proxy manage my TLS certificates as well, and Fabio seemed a little more service-level load balancing-focused, sort of like a Kubernetes Service. That may be mistaken, as I haven’t actually experimented with it very much. But it seems like a solid project and a potential replacement for the role that something ilke Caddy fills :+1:

5

I am going your way with caddy, but I am trying to make it even simpler: firecracker vms, redis as KV (instead of consul) and caddy+caddy_cgi to spin it off.

6

Fabio is cool, but I’ve found there’s a long-standing websockets buffering bug that means that the logs function in the Nomad UI doesn’t work.

Might use this approach to replace Fabio myself!

7

Informative post. consul-template is powerful.

Worth mentioning Caddy can generate reverse proxy configs dynamically from DNS SRV records, too. This can be made to work with Consul’s DNS server:

*.lab.mysite.com {
    # ...

    map {host}                       {subdomain} {
        ~(.*)\.lab\.mysite\.com$     ${1}       
    }   

    # Proxy services with the 'http-reverse' tag.
    handle {
        reverse_proxy {
            dynamic srv {
                name service.consul
                service {subdomain}
                proto http-reverse
                refresh 10s
            }
        }       
    }   

    handle_errors {
		# Do whatever you want here to return error info to clients
		# when a service is not found, etc.
    }   
}

This config assumes Consul is working as the system’s resolver. A consul agent can be running locally with a recursors config to pass other DNS requests upstream.

This doesn’t allow as much fine tuning as the templating approach, but at least the proto field enables filtering by a service tag.