Really enjoyed reading it, thank for taking your time for the write up!
One question I have is how did you install Caddy? Did you build from source? As you correctly saw the spikes when Go’s garbage collector was running, when building with Go > 1.19 you have the option to specify GOMEMLIMIT, which is a soft limit of RAM for when the garbage collector should run. As your target server has 4GB of RAM, my assumption would be, this would improve the performance quite a bit in the more CPU intensive tests.
On the other hand, I am very impressed by nginx almost never overstepping 32MB of memory used.