I expect the EspressoBin has good performance with OpenVPN. I ordered an EspressoBin in the initial Kickstarter, but have not needed to use it yet. (Given that the EBin a “dev” board, installing Armbian looks to be more work than installing Raspbian on an RPi. This has deterred me, at least temporarily. Friction, you know.)
ZeroTier (which is similar to OpenVPN) sells a network appliance based on the EspressoBin.
I’d like to specifically call out leming for his work on this board. While I helped answer questions and debugging, 99.99% of the work for support was handled via leming, even before the board was publicly accessible!
Nothing aside from a short ShieldsUp! scan, which didn’t find anything (ports cloaked). There’s so little running on the board that it’s pretty easy to enumerate what’s listening - I do have some forwarded ports to internal-only LAN hosts, but that’s about it.
I haven’t tried OpenVPN, but I would be curious about the performance. Although it’s a small CPU, it does have some crypto extensions that might aid with OpenVPN/tunnel performance. Here’s the lscpu output, which shows that the CPU does have AES and several checksum extensions, so OpenVPN performance may not be too bad.
[root@host ~]# lscpu
Architecture: aarch64
Byte Order: Little Endian
CPU(s): 2
On-line CPU(s) list: 0,1
Thread(s) per core: 1
Core(s) per socket: 2
Socket(s): 1
Vendor ID: ARM
Model: 4
Model name: Cortex-A53
Stepping: r0p4
CPU max MHz: 1000.0000
CPU min MHz: 200.0000
BogoMIPS: 25.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 cpuid
Hello, I have build the router, it’s a real nice project. It is running, but I am stopped on Part Five, at building ‘ipt-netflow-dkms-git’.
My command is:
yaourt -S ipt-netflow-dkms-git
but I encounter the following build error:
==> Building and installing package
==> ERROR: ipt-netflow-dkms-git is not available for the ‘aarch64’ architecture.
==> ERROR: An unknown error has occurred. Exiting...
==> ERROR: Makepkg was unable to build ipt-netflow-dkms-git.
I have read about the issue you encountered, but did not found how to get around the “not available for the ‘aarch64’ architecture.” error.
Did I miss something?
With all my thanks for this project,
Gaetan
1 reply
I’m glad to hear the build is running for you, that’s great!
I forgot to mention that the AUR PKGBUILD for ipt-netflow doesn’t currently list aarch64 as a supported architecture. If you edit the arch list to include aarch64 manually, the build should compile, install, and run just fine on the espressobin.
It’s been many months since I’ve started using netflow monitoring on the firewall at this point without any problems whatsoever, so I think aarch64 is a supported architecture for the module - I’ve just commented on the AUR package to request that aarch64 be added, so hopefully the maintainer does so in order to avoid making people add the architecture manually.
Thank you for your answer. After including ‘aarch64’ in the arch list, building the package went smoothly.
I followed your tuto to create the .conf files and loaded the ipt_NETFLOW module.
I got a warning: ‘ipt_NETFLOW: loading out-of-tree module taints kernel’, but no error.
I could not verify the content of the capture, because Arch Linux does not have a Logstash package. There is a Elasticsearch package, but I have still to look if it is the full stack (with Logstash) and find how to use it.
Anyway, I could verify with netcat that some data is sent to the port 2055 of my server:
netcat -u -l -p 2055 > netflow.capture
Update to my former post.
There is an Arch Linux Logstash package for x86_64 architecture. I didn’t found it at first, because I was searching for my home server, an ODROID-HC1, whose arch=armv7h.
So, after installing the ELK stack (Elasticsearch, Logstash, Kibana) on a x86_64 machine, and starting the services, I could resume the ‘Netflow Monitoring’ section of your tuto, and have those nice Kibana reports. I was really impressed to have all this working. Hopefully, you gave sufficient clues to know where to go.
Many thanks to you for having written this tuto.
About building ipt_NETFLOW DKMS module on Arch Linux:
For information:
Wow! its an amazing guidance and a beneficial blog .It is quite amazing and the way you explained it step by step it was fabulous.I have been looking for a walkthrough like it for a very long time but couldnt find a proper one and at last I finally found it and also it is amazing.You are a great blogger.williamjacket
Hi @EdenSajid, pretty much any modem will work with this setup, as long as the espressobin can get a DHCP address. At the moment I’m using this Netgear modem, which as been running for about a year.
@tylerjl Thank you so much for the tutorial. It’s been great playing with the espressobin! My current setup is practically identical to yours minus all the NETFLOW stuff. I’m noticing a huge reduction in speedtest when switching from my current pfsense router to the espressobin. I typically get 200down/11up with pfsense. With espressobin I’m getting 45down/15up.
I’ve gone over all the tcdevices and tcclasses and don’t see why it’s being limited. Any thoughts?
Thanks
1 reply
Linux alarm 4.18.14-1-ARCH #1 SMP Sat Oct 13 18:35:28 MDT 2018 aarch64 GNU/Linux
I followed your tutorial almost to the letter. I used http://wiki.espressobin.net/tiki-index.php?page=Boot+from+removable+storage+-+ArchLinux to setup the sdcard boot device.
I update the router about once a month, just to ensure all the relevant packages are kept current with upstream. So far the only breakages have been in kernel incompatibilities with the ipt-netflow module, but I think that’s only happened once so far - any Arch updates to shorewall, dnsmasq, etc. have been stable.
First let me thank you for that great blog post!
Over the weekend I tried to follow you guide but I came across a curious, at least for me, problem. Ip forwarding seems only to work if I start the network service after the shorewall service. On the clients when I try to traceroute any webaddress, im stuck at the router, after stopping and starting systemd-networkd.service the route completes. Is this normal behaviour and I just did something wrong, or did you take care that the network service starts after the shorewall?
1 reply
Hmm, I haven’t modified either the systemd-networkd service or the shorewall service. I believe that indicating the wan and br0 interfaces should be configured for IPv4 is sufficient - here’s my relevant files:
[root@router ~]$ cat /etc/systemd/network/br0.network
[Match]
Name=br0
[Network]
Address=192.168.1.1/24
IPForward=ipv4
IPMasquerade=yes
ConfigureWithoutCarrier=yes
[root@router ~]$ cat /etc/systemd/network/wan.network
[Match]
Name=wan
[Network]
IPv6AcceptRA=no
DHCP=ipv4
BindCarrier=eth0
IPForward=ipv4
[root@charon ~]#
As far as I know, this should set the interfaces into a forwarding state.
I read up on traffic shaping strategies for a long time before finally just settling on using eqhmcow’s strategy primarily because I felt like I understood the class choices and it seemed to fit my use case.
I suspect that latest-and-greatest traffic shaping algorithms (used to be codel, now cake) may achieve the same results, honestly. My understanding is that the tradeoff is router load, but after my time with the espressobin, I certainly think that it could handle the additional processing power.
My hope is that I can provide a revisit of my router setup after a while and spend some time benchmarking cake versus other options to verify whether it works, but if anyone wants to experiment, my understanding is that the module(s) are readily available from the Arch repositories/AUR.
Thanks for a nice post!
I did something similar - built my router using an Espressobin. I used Ubuntu rather than arch.
Works very nicely. Today I’ve had fiber based Internet service installed - 1Gb/s downsteam (replacing the meager 50-60Mb/s I had previously). Suddenly I hit a performance wall. The board goes to high CPU when I speedtest it, and hits a ceiling at ~125Mb/s.
Looking at what’s going on I see pppoe using very high CPU. Tried to upgrade rp-pppoe, but to no avail.
Any thoughts on that one?
EDIT: Typically after giving up on solving it and posting, you hit the solution. So I did. I’ve been using userland pppoe - and sure enough, it hits userland limits. Once I moved to kernel pppoe, Performance skyrocketed to what I believe is the ISP limit - at this moment, well over 500Mb/s.
1 reply
It seems I am also having speed issues as well with lan to wan traffic. Lan to Lan I am getting gigabit, from the Espressobin OS console outbound I am getting gigabit, but from a machine plugged into the lan to a machine hooked directly into the wan port I am only getting about a 3rd of a gig. Have you tried a bandwidth test on your router with something capable of gigabit speeds? I am getting these speeds from stock installs of Arch and Armbian, with no mods and no FW…
I saw doron’s comment, I do not believe the issue is pppoe on mine, at least I am not seeing that in the logs. Any ideas?
1 reply
@RickS - what I did is change my pppoe configuration template. It had a “pty” line calling /usr/sbin/pppoe with some flags, which translates to running pppoe as a userland process. (When I was on VDSL, it worked just fine; when I connected to FTTH, CPU went to the roof and performance was blocked.)
Instead, I –
This makes pppoe use the kernel module. Performance upped 5-7 times once I did that.
Hope this helps!
@Andy : I’m now getting slightly over 500 Mb/s from lan1 to wan routed by the EB. But, read on.
There’s another, more “subtle” performance limit, which is based on the architecture of the EB.
As you probably know, the board has a SOC with one 1Gb/s port; and an Ethernet switch (topaz), with 4 1Gb/s ports, one of which internally connected to the SOC.
Now, as long as LAN switching can be done inside the switch, you can get full 1Gb in / 1Gb out performance between, say, wan and lan1. The board, and, mainly, the kernel code, are very smart in offloading functionality into the switch. So stuff like basic routing, and even some iptables filtering, - can mostly be offloaded into the switch so there’s little performance impact.
But: when you consider traffic between two of the switch’s ports, with processing that must be done at the CPU level, - all that traffic needs to flow into the CPU and then back into the switch. This will now hit the limits of both the internal port (each packet needs to go over it twice - in and out), and the kernel which now needs to deal with double the packet handling interrupts (on a single core). While the port is full duplex, this translates to high load processing those packets, and on Linux you may see Soft IRQ overload.
Case in point (mine): I needed to terminate the ISP PPPOE tunnel on the EB. This means that the board needs to perform both PPPOE tunneling and point-to-multipoint NAT (aka masquerading). These can’t be offloaded to the switch chip; hence all my wan port traffic flows into the CPU and then down to lan port. High interrupt load on CPU. Net effect: tops at somewhat over 500Mb/s.
If you will terminate your tunnel (and do your inevitable NATing) outside of (i.e. in front of) your EB wan port, you may probably get much closer to the 1Gb/s theoretical limit.
Hope this helps!
1 reply
@doron - I actually don’t think I have the same issue as you, also I wouldn’t mind 2/3rds of a gig’s performance, but its far lower. Also I have a fiber box that I connect my router to, so I am not using PPPOE, and just to be sure, I made sure it wasn’t even installed. I did some searching, any my problem accurately matches this guys issue:
I’ve played around with the IRQ, but I cant get it to balance, its always running on either CPU 0 or 1, and at nearly 100% during a load, which is probably why its at 1/3rd performance. Again, not even sure if that is the issue, I just don’t know how to go about troubleshooting this…
@m7mdcc the STL file from the picture of my router is this one from Thingiverse.
I’ve also printed this case which houses a drive; as I’ve started experimenting running the espressobin with a drive attached as the root volume instead of an SD card and the setup works fairly well.
Does anyone have any advice for what to do in case of a power outage? My espressobin was running great until a storm rolled through and killed the power for a few hours. My UPS isn’t large enough to run my network forever and eventually it all came down while I was away at work. I came back to a corrupt sd card and my attempts at fixing it resulted in kernel panics. I’m hoping I can recover my config and set it all back up again. Is there a clever way to shutdown cleanly?
1 reply
APC makes an Arch linux package for a clean shutdown!
What version of Espressobin did you do this on? My guess that your build was the V5 just because of the timing of your post … I want to replicate what you have done. It seems to me that the only major difference between V5 and V7 is DDR3 => DDR4 … I think I will purchase the V7 Espressobin.
BTW, this was an amazing blog post! I have been reading about building my own firewall for a while and you seem to be the best resource for doing that 
Yep! This was on a V5. I think a V7 would be a great option for a router; the additional CPU horsepower means that you could probably use more intelligent QoS schedulers like CAKE without hitting any bottlenecks.
Hi @Morta! Here’s the relevant parts of my dnsmasq.conf. I have a couple of extra cname= and similar entries for some other names I have setup note that there are a few things specific to my setup here:
5 since I have a few reserved static IPs.conf-dir= option is also just optional, I have some dynamically-generated files there.server=/consul/127.0.0.1#8500 lets any requests for foo.service.consul resolve from any host in my LAN.domain-needed
bogus-priv
server=/consul/127.0.0.1#8600
no-hosts
domain=<my domain>
expand-hosts
conf-dir=/etc/dnsmasq.d
interface=br0
dhcp-range=192.168.1.5,192.168.1.250,255.255.255.0,24h
dhcp-option=option:router,192.168.1.1
address=/router/192.168.1.1
dhcp-boot=ipxe.a56af4e6a9a9.pxe
enable-tftp
tftp-root=/srv/ftp
Hi Tyler.
Very good article indeed. I am already running a router on arch linux myself on an APU1D4 and custom intel motherboard with dnsmasq,unbound and shorewall.
I came across your article while searching for a way to visualize network data.
I tried the instructions on elasti.co but I am unable to set it up and getting nowhere.
Would it be possible for you to post how you set it up on arch linux? The second machine where I want to send netflow data is also running arch.
Than you.
I need a dual-WAN solution. Think it’s possible to use this as a router, but just treat the “wan” interface as LAN and use eth0 and eth1 for my PPPoE connections?
Is there any reason why that wouldn’t work? Otherwise I suppose I could buy a USB-to-Ethernet adapter and use it for the second WAN port
1 reply
Hi @fabiolourenco, I think I mention why not pfsense in the post, but the short version is that I wanted Linux since I would be best able to administer it instead of a *BSD and had some additional features I wanted to experiment with in the Linux kernel.
Hi @manpreet! In short terms, here’s what I’m doing in order to visualize my network data:
On my Arch Linux ARM-based router, I’ve installed
the ipt-netflow-dkms-git AUR package. I then load that kernel module on-demand by including the following lines in /etc/shorewall/start:
run_iptables -I INPUT -j NETFLOW
run_iptables -I FORWARD -j NETFLOW
run_iptables -I OUTPUT -j NETFLOW
return 0
You configure the destination for netflow packets in /etc/modprobe.d/ipt-netflow.conf, mine looks like this:
options ipt_NETFLOW destination=127.0.0.1:2055 protocol=5
This is because I run a dynamic reverse proxy on the router to send netflow packets wherever Nomad is currently running logstash, but the tl;dr is that you point the module at wherever you’re running logstash.
The logstash config is simple due to the netflow module:
http.host: "0.0.0.0"
modules:
- name: netflow
var.elasticsearch.hosts: elasticsearch.service.consul
var.kibana.host: <my kibana hostname>
var.kibana.scheme: https
The steps to get Logstash connected to Elasticsearch is outside the scope of the guide I’ve written here. Does that make sense/are there other parts I could clarify?
@jrutley I think that use case should be doable. I haven’t tweaked my setup very comprehensively, but at the very least, each of the three network interfaces do show up distinctly. I’m not sure what might be implied by the fact that most are of the form @eth0, though.
$ [root@charon network]# ip l | grep -A1 eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1508 qdisc mq state UP mode DEFAULT group default qlen 1024
link/ether xx:xx:xx:xx:xx:xx brd ff:ff:ff:ff:ff:ff
--
4: wan@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000
link/ether xx:xx:xx:xx:xx:xx brd ff:ff:ff:ff:ff:ff
5: lan0@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br0 state UP mode DEFAULT group default qlen 1000
link/ether xx:xx:xx:xx:xx:xx brd ff:ff:ff:ff:ff:ff
6: lan1@eth0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue master br0 state LOWERLAYERDOWN mode DEFAULT group default qlen 1000
link/ether xx:xx:xx:xx:xx:xx brd ff:ff:ff:ff:ff:ff
Great write-ups, I ended up on a PC Engines APU4 but with a similar way of thinking.
Even if I love Arch and has used it as a daily driver for almost a decade now, I wouldn’t dare running it on a router (the systemd migration mess still burns) - great to see that it’s working fine for you!
With regards to OpenWRT (which is what I am running), If you build your own OpenWRT image from source it’s actually a pretty nice flow to have a squashfs root with a writable ext4 overlay (make menuconfig to select packages and configure kernel before compiling). That way you can “reflash” and know exactly the state of the system, apart from the config files and user directories which explicitly get copied over between reflashes.
You can still manually opkg install packages and poke around everywhere, so I find it a pretty good workflow so far to test things that way and then incorporate any new/removed packages in a new image (opkg upgrades and conflict resolution leaves a lot to desire though - I started out on a fully writable ext4 root, which quickly became a hot mess of conflicts and annoyance and anyone attempting this should know there’s good reasons the maintainers recommend against this and “opkg upgrading” by installin new versions of packages on a running system). The downside is the need to reflash every time you need security updates, of course.