There have been improvements to our Quality of Service (or QoS) which have made me very excited.

Our QoS sometimes was a bottleneck. Enabling it could cut your bandwidth in half if you were unlucky. That normally was not a problem for larger users of IPFire, because if you are running a 1 Gigabit/s connection, you would not need any QoS in the first place, or your hardware was fast enough to handle the extra load.

For the smaller users this was, however, becoming more and more of a problem. Smaller systems like the IPFire Mini Appliance are designed to be small (the clue is in the name) and to be very energy-efficient. And they are. They are popular with users with a standard DSL connection of up to 100 Megabit/s which is very common in Germany. You have nothing to worry about here. But if you are lucky to have a faster Internet connection, then this hardware and others that we have sold before might be running out of steam. There is only so much you can get out of them.

In the test case for this, a six year-old IPFire Eco Appliance with an Intel D-525 processor with two physical cores and Intel Hyper-Threading was connected to a 400 MBit/s Internet connection. Without QoS enabled that bandwidth was easily handled with a CPU load of around 20%. After QoS was enabled, the throughput collapsed to around 120 MBit/s at less than 50% CPU load. Clearly that drop seemed to be too high and there was still some CPU cycles left to use.

And that is where the search began...

Finding the bootleneck

IPFire is very tuned. We have spent so many development hours over the years to get everything optimised as much as possible. That is why there were not too many things left to review again.

The QoS is split into three parts: The first one is to classify packets when they arrive. We will look if any of the configured rules match and if so mark the packet to belong to a certain class. Packets are then being sent through the whole firewall ruleset and potentially allowed to pass or being dropped. After the firewall has made the decision to which interface the packet is being sent, it is being enqueue in a so-called qdisc until there is free bandwidth and it can be sent.

It turned out that there was space for improvement in all those steps.

iptables is now able to decide into which class to sort a packet in one go. Before this used to be a two step process where the packet was being marked first and then later on move into the correct class. Since this can now be done in one step, this is bringing us a little advantage, which unfortunately is not noticeable (less than a percent increase), but it makes our code cleaner because we had some problems with interference of the Intrusion Prevention System and NAT which need to mark packets, too.

ToS

The second improvement is to drop support to changing the ToS (Type of Service) bits. Those would mark if a packet should be forwarded with lowest latency, cheapest route and some others more. Those are however not evaluated by any ISP and it is just a waste of CPU cycles to mark those packets us such. The whole checksum of the IP packet needs to be recalculated which adds a lot of overhead for a useless operation.

Applications like VoIP phones and Skype set these bits on the client, so rules can still be used to find this traffic when it is coming from the local network and put it into the right class. That is very handy for crystal-clear VoIP calls!

Breakthrough!

That again improved throughput only very little. The biggest break-through was the realisation that one processor core was more busy with waiting for other cores to handle a packet than processing packets itself. It wasted time on waiting, which is not great when you do not have the fastest processor anyways.

To process incoming packets - which only ever was the bottleneck - we send them through a device called Intermediate Queueing Interface (IMQ). That is a small driver which has never made it into the Linux kernel and was patched into the IPFire kernel by us. It always worked flawlessly, but evidently was not good enough to be merged into the mainline kernel. After reading this post, you know why.

More advanced appliances use multiple queues when they receive packets from the network. Cheaper network adapters do not support this, but the better ones do. That way, each processor core can handle packets from their own queue and that means you do not only use one processor, but all of them at the same time.

Now, TCP connections have packets in an order. Therefore there needs to be synchronisation between the processor cores to align data in the right order to run the layer 7 filter over it. For that, processor cores need to talk to each other and that is normally a very expensive operation. It is extremely slow on Intel Atom processors, but rather fast on Intel Xeon processors. I do not need to get into detail on how that relates to power consumption.

Mostly the QoS does not really care about TCP. It works on layer 3 which is only IP. IP connections are stateless and therefore less synchronisation would allow us to process more packets.

The IMQ driver has 38 mentions of the word "lock" in its source code. It uses them excessively and has always been criticised for its poor handling of SMP systems. The implementation that is coming with the Linux kernel - and is working slightly differently - only has two mentions, hence, uses a lot less locking.

I will no longer bore you with the technical detail. This post is probably long enough already. I am bad with keeping things short. If you are really keen have a look at the patchset.

Let's fast-forward to the results

The same appliance is now forwarding packets with a whopping 390 MBit/s and has CPU utilisation of around 40-50%. CPU usage therefore has not gone down, but throughput has reached the maximum it could possibly be. Goal achieved!

Naturally IPFire is able to shift more than 400 MBit/s of traffic now, but this was the real-world test. In a lab I am being able to saturate a 10G link with almost no noticeable change in CPU utilisation on an IPFire Enterprise Appliance which is brilliant, too.

Thank you for your help - please donate!

All of this was done over two intense afternoons. Sitting down with a cup of tea and analyse something until you find the root cause of it. Garnishing it all with a nice improvements to the web UI and there you are: IPFire has massively improved its performance on small systems and even on larger ones.

All of this will come with IPFire 2.23 - Core Update 137 which will be released in a couple of weeks.

This would not have been possible if I didn't have access to the hardware and Internet connection. Thank you very much for your support - you know who you are. If you want to support our project, too and want to help me paying for my tea, please donate. It is very important to be able to work on things like this!


Today, we have updated IPFire on AWS to IPFire 2.23 - Core Update 136 - the latest official release of IPFire.

This update includes security fixes for OpenSSL and the Linux kernel, an updated Perl, and of course many other fixes throughout the whole system.

AWS

We are very happy that from week to week, we are gaining more customers for IPFire in the cloud - where you now can manage your network just as you do it in your own data center.

In contrast to Amazon’s own features, IPFire is easier to manage, performs just as well, but brings you even more features like standard IPsec VPNs, OpenVPN for on-the-road connectivity to the cloud, Intrusion Prevention for your cloud servers, detailed logging and reporting and many more features.

Try it out today for free!

There is a detailed installation guide available which helps you setting up your cloud correctly for IPFire.

How to update?

For all customers that are already running on the latest image, there is nothing to do here but to make sure that you have all updates installed on your instance.

Click here to go to IPFire on AWS


IPFire 2.23 - Core Update 136 released

by Michael Tremer, October 10, Updated October 13

This is the official release announcement for IPFire 2.23 - Core Update 136. A new update packed with loads of security fixes, bug fixes and a couple of important new features.

Please donate to help our developers and keep bringing you new features. Thank you, it means a lot.

OpenSSL 1.1.1d

This update ships the latest update of the OpenSSL library which has received some important fixes in its latest release:

  • CVE-2019-1547: With custom elliptic curves, timing attacks were made possible again. This is of very low risk in IPFire, since we are not using any custom curves.
  • CVE-2019-1549: Forked processes could have shared the same seed for their random number generator which is being fixed in this one by mixing in a high precision timer.
  • CVE-2019-1563: Another padding oracle for large PKCS7 messages

All of these are classified as "low severity". However, we recommend to install this update as soon as possible.

Perl 5.30

Arne has been busy and been working on replacing Perl with the latest stable version. This requires that loads of applications that use Perl - like our own web user interface - have to be shipped again as well as many add-ons. Hence this update is rather large.

GeoIP

Since Maxmind is no longer publishing their GeoIP database in the original format, but unfortunately not providing any good bindings for the new release, we have only had an outdated version of the database that we made available in IPFire.

There is now a script that converts the current data into the old format which allows us to provide a recent database again.

This database is however only being used for showing the country flags on the web UI. GeoIP blocking uses a database in a different format and therefore always has recent data to only block the right things.

Misc.

  • The firewall has a limit for log messages so that flooding the firewall with packets won't cause a Denial-of-Service by filling up the hard drive with gigabytes of logs and also to not starve on write operations. This limit was however very low for modern standards and has therefore been increased to 10 logged packets per second. That will ensure that we won't drop a packet without logging it.
  • Updated packages: apache 2.4.41, bind 9.11.10, clamav 0.101.4, dhcpcd 8.0.3, knot 2.8.3, logrotate 3.15.1, openssh 8.0p1, patch 2.7.6, texinfo 6.6, unbound 1.9.3, usb_modeswitch 1.5.2
  • logwatch and logrotate could conflict when running at the same time. This has been changed so only one of them is running at the same time.
  • Log messages for DMA, the IPFire mailer, and Postfix are now shown on the web UI
  • The toolchain now ships a compiler for Go

Add-ons

  • Updated packages: freeradius 3.0.19, haproxy 2.0.5, monit 3.25.3, postfix 3.4.6, spamassassin 3.4.2, zabbix_agent 4.2.6
  • dnsdist has had its limit of open connections increased to work better in bigger environments
  • tor: A permission problem has been fixed so that the web UI can save settings again
  • wio: The RRD files will now be included in the backup as well as various UI improvements have been done

Please reboot!

This update needs a reboot of your IPFire system.