This blog post is about benchmarking Caddy against Nginx and their respective performance metrics as reverse proxies. Be forewarned: I was very thorough and there are tons of graphs and tables in here. I didn’t want to make any mistakes! Nobody is allowed to make mistakes on the Internet.
Really enjoyed reading it, thank for taking your time for the write up!
One question I have is how did you install Caddy? Did you build from source? As you correctly saw the spikes when Go’s garbage collector was running, when building with Go > 1.19 you have the option to specify GOMEMLIMIT, which is a soft limit of RAM for when the garbage collector should run. As your target server has 4GB of RAM, my assumption would be, this would improve the performance quite a bit in the more CPU intensive tests.
On the other hand, I am very impressed by nginx almost never overstepping 32MB of memory used.
My system under test was a NixOS 22.05 machine, which builds Caddy using this package. Nix is sort of funky to read but it’s built with go 1.17. Maybe in the future I can tweak the build environment to see what that kind of change might cause, but I’d certainly be curious as well.
Thanks for sharing your benchmarks. In the past I’ve done some Nginx vs Caddy benchmarks and noticed as you add more response headers to requests, Caddy’s performance drops more than Nginx does (see links below). As as I’ve been always mindful to test with real world used response headers/equivalent for both web servers.
So I’m curious if in your proxy testing for latest Caddy versions, does the same performance overhead happen with Caddy still exist as you add more response headers?
Performance measurements that include response headers is something I haven’t tested yet - since I published this research I’ve received lots of helpful input about some additional variables to test, and headers are one of those variables that I think would be helpful to include. I’ve recorded your suggestion here if and when I do another round of tests
Signed up/in just to say thank you for your incredibly thorough and detailed piece. Although it is now two years old, it still helps educate and explain. I, too, love Caddy, having used nginx and apache before it for many years. This helps me understand and gives me an improved perspective to know how to ask the right questions of performance of my reverse proxy setup. So again, thank you! GREAT work.