Today, we are releasing a brand new version of IPFire: 2.27 - Core Update 161. Amongst a huge performance improvement for the Intrusion Prevention System, it comes with a brand new kernel and various security and bug fixes.

Before we talk about what is new, I would like to ask you for your support. IPFire is a small team of people and like many of our open source friends, we’ve taken a hit this year and would like to ask you to help us out. Please follow the link below where your donation can help fund our continued development: https://www.ipfire.org/donate.

Please note, that this update will reconnect any PPP connections and we recommend performing a reboot after the update has been installed.

Boosting Intrusion Prevention System Performance

The most notable change in this update is a large increase of throughput of the IPS. It can now decide to no longer see traffic from a certain IP connection and tell the kernel to bypass it. That removes all overhead for these connections and therefore increases throughput.

On systems like the Lightning Wire Labs Mini Appliance which comes with four CPU cores each at 1 GHz clock speed, it boosts throughput from about 120 MBit/s on full CPU load to 1 GBit/s on about 20% load on one CPU core for this type of connection. This releases more CPU time for scanning other traffic and allowing this device being properly used on connections with more than 100 MBit/s throughput.

For this change, a lot of work around the QoS and VPNs were necessary because of touch points in the firewall engine. Here, we were also able to tidy up code and make the system more efficient.

Fast Flux Detection in Web Proxy

This update brings Fast Flux Detection as introduced by Peter.

Updated OS Kernel

The IPFire kernel is now based on Linux 5.10.76 and various configuration changes have been made:

  • Hardening of stack variables: All of those will now be zero-initialised to avoid any information leak inside the kernel's memory space
  • TPM hardware is now being used as a source for entropy if available
  • The kernel will now wake up more often in order to keep packet forward latency down and make the system more responsive.
  • Some debugging/overhead functions have been disabled for slight performance gains

Misc.

  • Python 2 has been removed from IPFire with this release
  • IPFire now supports ExFAT
  • Logwatch now includes status of software RAID configurations
  • Regressions in the disk utilization stats due to a change in iostat(8)'s output have been fixed
  • After launching an update, the Pakfire page did not correctly show the locked state
  • The web proxy will now always hide its version number due avoid any information leaks
  • Support for FriendlyARM NanoPI R2S has been added
  • Updated packages: apache 2.4.51 fixing CVE-2021-42013 introduced due to an incomplete fix for CVE-2021-41773, curl 7.79.1, dosfsutils 4.2, GD-Graph 1.54, gd 2.3.3, iproute2 5.14.0, perl-GD 2.73, strongSwan 5.9.4

Add-ons

  • Tor will now use any hardware acceleration for cryptographic operations if available
  • Updated packages: 7zip 17.04, cups-filters 1.28.10, Ghostscript 9.55.0, Git 2.33.1, htop 3.1.1, krb5 1.19.2, monit 5.29.0, nano 5.9, pcengines-apu-firmware 4.14.0.4, shairport-sync 3.3.8
  • avahi's and minidlna's confguration is now correctly backed up and restored on updates

A new update is available for testing: IPFire 2.27 - Core Update 161. It comes with a huge performance improvement for the Intrusion Prevention System which allows it to deliver excellent throughput even on smaller hardware. On top of that come a brand new kernel and various security and bug fixes.

Please note, that this update will reconnect any PPP connections and we recommend performing a reboot after the update has been installed.

Boosting Intrusion Prevention System Performance

The most notable change in this update is a large increase of throughput of the IPS. It can now decide to no longer see traffic from a certain IP connection and tell the kernel to bypass it. That removes all overhead for these connections and therefore increases throughput.

On systems like the Lightning Wire Labs Mini Appliance which comes with four CPU cores each at 1 GHz clock speed, it boosts throughput from about 120 MBit/s on full CPU load to 1 GBit/s on about 20% load on one CPU core for this type of connection. This releases more CPU time for scanning other traffic and allowing this device being properly used on connections with more than 100 MBit/s throughput.

For this change, a lot of work around the QoS and VPNs were necessary because of touch points in the firewall engine. Here, we were also able to tidy up code and make the system more efficient.

Fast Flux Detection in Web Proxy

This update brings Fast Flux Detection as introduced by Peter.

Updated OS Kernel

The IPFire kernel is now based on Linux 5.10.76 and various configuration changes have been made:

  • Hardening of stack variables: All of those will now be zero-initialised to avoid any information leak inside the kernel's memory space
  • TPM hardware is now being used as a source for entropy if available
  • The kernel will now wake up more often in order to keep packet forward latency down and make the system more responsive.
  • Some debugging/overhead functions have been disabled for slight performance gains

Misc.

  • Python 2 has been removed from IPFire with this release
  • IPFire now supports ExFAT
  • Logwatch now includes status of software RAID configurations
  • Regressions in the disk utilization stats due to a change in iostat(8)'s output have been fixed
  • After launching an update, the Pakfire page did not correctly show the locked state
  • The web proxy will now always hide its version number due avoid any information leaks
  • Support for FriendlyARM NanoPI R2S has been added
  • Updated packages: apache 2.4.51 fixing CVE-2021-42013 introduced due to an incomplete fix for CVE-2021-41773, curl 7.79.1, dosfsutils 4.2, GD-Graph 1.54, gd 2.3.3, iproute2 5.14.0, perl-GD 2.73, strongSwan 5.9.4

Add-ons

  • Tor will now use any hardware acceleration for cryptographic operations if available
  • Updated packages: 7zip 17.04, cups-filters 1.28.10, Ghostscript 9.55.0, Git 2.33.1, htop 3.1.1, krb5 1.19.2, monit 5.29.0, nano 5.9, pcengines-apu-firmware 4.14.0.4, shairport-sync 3.3.8
  • avahi's and minidlna's confguration is now correctly backed up and restored on updates

Thanks to libloc, the free & open source location database, IPFire comes with an accurate, trustworthy database for mapping IP addresses to countries and Autonomous Systems, and vice versa. This allows us to introduce a new feature: Proactive detection of Fast Flux setups, which are commonly used by ne'er-do-wells for hosting questionable and malicious content on compromised machines around the world, switching from one infected PC, IoT device, or router to another within minutes.

To the best of our knowledge, this is a unique feature. Contrary to other security mechanisms such as AV scanners, which are often lagging behind, it detects malware, phishing, C&C servers and other nefarious things proactively - before any threat intelligence source in the world even knows about them. Even better, measurements done so far indicate it comes with a near-zero false positive rate in productive environments.1

If you are using IPFire's built-in web proxy, all you need to do is to tick a checkbox, hit the "save and reload" button at the end of that page, and you're done.

To compensate the rather simple looking screenshot, this blog post explains what Fast Flux hosting looks like, how it is used by cyber criminals, and how IPFire detects it. If you are in need of some tea or coffee, it is now time to make it. Ready? Here we go...

What is Fast Flux hosting?

Fast Flux is a way of hosting content on a highly volatile network of usually compromised machines across the globe. Instead of returning one or two relatively static IP addresses for a FQDN, as it is commonly done, a FQDN being in use for Fast Flux hosting returns a bunch of IP addresses, pointing into different countries and Autonomous Systems.

Replies to a DNS query come with a Time to Live (TTL), telling a DNS resolver how long it should cache them. While legitimate services often have a TTL within the range of hours or a day, Fast Flux FQDNs come with a significantly smaller one: Some of them propagate a TTL of half an hour, while others shuffle the IP addresses returned back every minute. Therefore, you'll end up talking to a different machine every time.

In theory, there are few legitimate use-cases for this. The author, however, never observed a Fast Flux hosting setup that had even a patina of legitimacy. In fact, they are an important utility for cyber criminals, especially because they are very resilient against take-down attempts. With ten new IP addresses coming in every few minutes, even most sophisticated and capable anti-abuse actors have to give up.

Most Fast Flux setups serve malicious content, ranging from malware distribution via phishing sites to C&C servers. Being exposed to detection and shutdown, infected systems often do not host valuable data such as collected login credentials themselves, but serve as proxies for connecting to the cyber criminals' backend infrastructure - which is often hidden behind another Fast Flux network, located at bulletproof ISPs or operating out of stolen IP space.

Although they are not suitable for applications requiring persistent, low-latency connections, some Fast Flux networks serve non-technical threats as well, such as marketplaces for stolen credit card data, fake pharmacy shops, warez download portals, or CSAM.

How does IPFire detect Fast Flux setups?

Having only a FQDN, a bunch of IP addresses resolved and perhaps a few metadata at hand, it is pretty hard to spot a Fast Flux network with a sufficient level of confidence. Very low TTLs are used for legitimate purposes as well (take a look at www.youtube.com, for example), and IP addresses obviously not being part of the same /24 or /16 chunk is not a very reliable criterion either.

Querying IP reputation services or RBLs might catch some incidents, but relies on information on infected machines being available in near-realtime. Given the volatility of such setups, false negatives are highly likely.

But Fast Flux hosting comes with one characteristic sticking out like a sore thumb: It has an extremely high ASN diversity, since IP addresses are most likely to trace back to many different ISPs, hence ending up in different Autonomous Systems - which have different Autonomous System Numbers (ASNs).

To be fair: High-availability services might be hosted in two different data centers behind two distinct ASNs. People might have a backup DC - but not four or five of them. It simply does not make sense.

In terms of proactive detection, this is a Fast Flux network's undoing. And this is what we are able to measure in IPFire as well, since every installation comes with our location database, already having all network data needed at hand, queried within nanoseconds. We just need to make use of it.

Below is a snapshot of the DNS replies of two FQDNs hosted on Fast Flux setups, dated around 9 pm (UTC) on July 14th, 2021. Their anatomy, spreading across several different ASNs in several different parts of the world, is striking.

IP Address Country ASN AS Name
1.247.35[.]250 KR 9318 SK Broadband Co Ltd
88.158.247[.]38 RO 15471 S.N. Radiocomunicatii S.A.
109.102.255[.]230 RO 9050 TELEKOM ROMANIA COMMUNICATION S.A
110.14.121[.]125 KR 9318 SK Broadband Co Ltd
175.117.131[.]127 KR 9318 SK Broadband Co Ltd
180.69.193[.]102 KR 9318 SK Broadband Co Ltd
181.57.221[.]246 CO 14080 Telmex Colombia S.A.
189.238.217[.]39 MX 8151 Uninet S.A. de C.V.
190.218.32[.]60 PA 18809 Cable Onda
213.231.134[.]136 BG 38932 Rimex Ltd.
IP Address Country ASN AS Name
1.247.35[.]250 KR 9318 SK Broadband Co Ltd
37.75.44[.]24 MT 33874 Epic Communications Limited
84.40.106[.]91 BG 43561 NET1 Ltd.
106.241.4[.]103 KR 3786 LG DACOM Corporation
116.126.116[.]6 KR 9318 SK Broadband Co Ltd
175.117.131[.]127 KR 9318 SK Broadband Co Ltd
186.6.236[.]46 DO 6400 Compañía Dominicana de Teléfonos S. A.
211.244.109[.]130 KR 9318 SK Broadband Co Ltd
218.232.207[.]201 KR 9318 SK Broadband Co Ltd
218.233.73[.]201 KR 9318 SK Broadband Co Ltd

One might get curious over the geographic distribution of these two examples. This is because a countries density of compromised machines is often related to different factors, including socioeconomic ones. However, South Korean ISP "SK Broadband" clearly has a problem with tackling abusive activities originating from their networks.

Seeing the bigger picture: Can we get rid of Fast Flux networks altogether?

Most probably, the short-term answer to this is "no": As explained by Brian Krebs multiple times, miscreants are both very tenacious in compromising machines and creative in abusing them afterwards. Owners of IT equipment, however, often say something like: "There is no valuable data on this PC, why should I bother securing it?" As long as they do not see the threats of an infection that emerge to the internet community, we will most likely continue to observe Fast Flux setups in the future.

As always, having both a good firewall ruleset and IPS configuration in place is vitally important to keep your clients behind IPFire secure. Using the web proxy, you do not have to worry about Fast Flux setups anymore, either.

Want an extra benefit? How about detection of selectively announced networks?

Especially in the spamming business, miscreants sometimes do not announce their networks globally, but only to a few ESPs (e-mail service providers) they aim at. That way, security vendors are less likely to notice, hence reducing the risk of disruption to the planned spam campaign.

In addition to deny access to a FQDN having an abnormally high AS diversity, you can also prevent your clients from reaching a website that is not hosted on a globally announced IP network - there simply is no legitimate reason for this.

This also applies to BGP announcements being very new: Would you like to visit a website being hosted on a network that just bursted into life a few hours ago? Why can't its operator wait a few days to ensure the network is known globally?

The author once observed such a setup (phishing mail containing a link to an AS not announced globally, but only to the victim's organisation) in conjunction with industrial espionage. Even if this is not a threat your network is exposed to, blocking access to destinations without a global BGP announcement might shield some dirt from it.

Both this and the Fast Flux detection will be part of the upcoming Core Update 161. Please donate if you like what we are doing. Should you want to use our location database outside of IPFire, further information on it is available here. Feel free to drop a line on the mailing list, if necessary.


  1. If populated, the list of allowed domains configured in the URL filter section will be used to bypass detection hits for these FQDNs. Should you encounter a false positive, just add the domain to the list, and reload the web proxy. 

  2. Source of the image above: Brian Krebs: The Scrap Value of a Hacked PC, Revisited