I am wanting to test microservices and web services and other things in a closed environment at my homelab which is currently 12 nodes (old dell servers mostly) but I also want to expose them to Internet as securely as possible and make use of my 5 static IP address and dozens of domain names I own. I have been struggling to get my setup just right so that I can serve beyond the lab lan side to actually sever to the Internet wan side.
I have OPNsense as a full VM sitting on top of a 12 node proxmox cluster. I configured OPNsense to be HA and live migrate to node2 if it fails on node1 and I have network cards that will switch over by default to passthrough data should the server node1 or node2 fail (power loss) or receive command to do so. (Silicom 4port managed nic cards in each).
Currently I have ISP–>eno1[labnode1] —>eno2 --(LAN network)…[nodes1-12]
On this node1 I have dedicated one nic to uplink ISP connection. I have 6 more nic ports I can play with as needed and 4 of them are failover bypass nic(would need to work in pairs) thus if enp1x0 drops it routes out of enp1x1 to another machine if I want it to… and if I were to add redundant ISP connection of some sort - maybe my extra cell phone with unlimited 4g data… I could use it into the other 2 ports on that card… that would fail over to my second machine (node server).
Forget about all the points of failure and focusing on just OPNsense on top of my proxmox cluster - > I have been having a hell of a time trying to get letsencrypt certs as well as Caddy or Nginx to work correctly.
Any tips or tricks?
I have kids minecraft server, immich, nextcloud, plex, and a few other services in addition to some testing web servers and locally hosted database servers all on the proxmox cluster on the lan side that see and talk to eachother fine… I just cant seem to get out past much less make them routable from Internet to the correct lan side ip and port without hard coding in the opnsense firewall rules various NAT rules.
I am learning - so don’t beat me up too badly here.
How can I make the best of my setup - route specific web requests to the correct VM on my cluster using Caddy or Nginx on top of OPNsense (plugin) and make use of my domain names? I have several domain names and subdomains with A records pointing to the IP of my server with opnsense running on it that acts as a gateway and firewall of sorts for all the other nodes connected on the lan side.
The next step has just pestered me because I can’t seem to get it configured right to pass the traffic from Ineternet side WAN side to the LAN side with any proxy plugin. It should be possible - and I am guessing it is a simple issue with my settings I did not do… but there is the proxmox layer to contend with too.
Lost dad would love some help here.