Thank you for this write-up. I have nomad/consul/vault running “distributed system” on my 3 raspberry PI with SeaweedFS.
Have you looked Fabio? It seems to be specifically designed instead of caddy in such setup - I was excited about caddy, but then I had a bad experience when tried to deploy it on somebodies else jetson nano remotely.
I looked at Fabio briefly, but I was very keen on having my reverse proxy manage my TLS certificates as well, and Fabio seemed a little more service-level load balancing-focused, sort of like a Kubernetes Service. That may be mistaken, as I haven’t actually experimented with it very much. But it seems like a solid project and a potential replacement for the role that something ilke Caddy fills 
I am going your way with caddy, but I am trying to make it even simpler: firecracker vms, redis as KV (instead of consul) and caddy+caddy_cgi to spin it off.
Fabio is cool, but I’ve found there’s a long-standing websockets buffering bug that means that the logs function in the Nomad UI doesn’t work.
Might use this approach to replace Fabio myself!
Informative post. consul-template is powerful.
Worth mentioning Caddy can generate reverse proxy configs dynamically from DNS SRV records, too. This can be made to work with Consul’s DNS server:
*.lab.mysite.com {
# ...
map {host} {subdomain} {
~(.*)\.lab\.mysite\.com$ ${1}
}
# Proxy services with the 'http-reverse' tag.
handle {
reverse_proxy {
dynamic srv {
name service.consul
service {subdomain}
proto http-reverse
refresh 10s
}
}
}
handle_errors {
# Do whatever you want here to return error info to clients
# when a service is not found, etc.
}
}
This config assumes Consul is working as the system’s resolver. A consul agent can be running locally with a recursors config to pass other DNS requests upstream.
This doesn’t allow as much fine tuning as the templating approach, but at least the proto field enables filtering by a service tag.