Attention! This is a long post. Please read it, because it’s important to you if you use ARM-based hardware. And you do. I’m sure. This post may get a bit ranty, but I think this is the only way to get the message out loud and clear. Despite that, it gives you insights about ARM development from a the IPFire project’s point of view.
ARM, the architecture, is everywhere. It powers our mobile phones and there are more and more boards that are able to run Linux – or more specific: IPFire.
IPFire supports ARM for several years now and we believe that is has a lot of advantages over the x86 architecture. Of course there are a lot of disadvantages as well, so please don’t misunderstand: ARM is not better than x86 – x86 is also not better than ARM. They are both great in one way or an other.
This post is not about the technical differences. I would like to talk about other problems the IPFire project has been facing that are related to the ARM architecture and the board manufacturers and SoC vendors. This is quite a big issue for us and we feel concerned about the future of ARM, unless some things change.
Board Manufacturers / Distributors
At first it has been a big problem is to get your hands on actual ARM hardware. Except the Raspberry Pi Computer, which I don’t count to the proper boards, it was sometimes hard to order them. Either there was no distributor in Germany or the EU or they were out of stock. Of course this chain of distributors, importers and merchants makes the products way more expensive. We paid sometimes more than twice the original price. So nobody can say ARM is cheap.
Since the Raspberry Pi Computer came out, there was an increasing demand for ARM hardware and the local distributors added more and more products to their stocks. Great!
What’s also great is that more ARM-based boards are developed. There are lots of development boards (which basically don’t come with a case) like Pandaboard and there are consumer products like the recently announced Utilite.
Intermezzo: An extra version of the Linux kernel per board
Some people may already know this, but for those of you who don’t: ARM machines don’t have an equivalent to the PC BIOS. Therefore, an operating system cannot figure out by itself which hardware there is and how to talk to it. Because of the well defined IBM-PC specification, an x86 kernel can talk to the BIOS and “ask” for that information.
The solution that has been used in the Linux kernel is to hard-code memory addresses of all the peripheral devices of the system. So what distributors like the IPFire developers need to do for every ARM board that is available on the market is building an extra kernel for each and everyone of it. And it’s even worse, because if the hardware changes even a little bit (like an extra 256MB memory), we have to ship an extra kernel for that.
There is nothing as a one-for-all solution. Obviously, we cannot do this for every board there is. We are currently doing it for only a few: Raspberry Pi, Pandaboard and Marvell Kirkwood-based devices.
So, more hardware it not really fun for us as a distributor, because it requires a lot of work, to support an extra hardware platform or board. It’s not that we need to do just one giant modification of the system that’s supposed to run on the hardware, but rather an ongoing process of maintenance – sometimes easier, sometimes harder. We will get to that later…
In addition to the work, we have to put in, we need the actual hardware for testing. Adding support for hardware cannot be done by guessing if something works. Doing it that way will never get the job done and requires some expertise of the person who is doing the actual testing. There are so many thing that could go wrong.
But of course we don’t have the money to buy all the ARM boards and we don’t have the time to support them all. We also don’t really feel to do it because we don’t get ANY support from the hardware manufacturers.
What do we want from them? Because as aforementioned, we need to get our hands on the hardware. There are two options in what way they could help us. Before we go into that, I would like to point out that we feel that the manufacturers who make money out of selling their products have both the responsibility and the resources to do software development, which in turn results in selling more units, because hardware is actually only worth what’s running on it.
Here are the two options:
Option #1: Send us the hardware
In order to get IPFire running on your hardware, please send it to us. We need it to get the job done. We don’t take any money for doing the work – we do it for free.
As an example: We have recently written to CompuLab, who is a company from Israel that announced Utilite, a nice consumer-friendly ARM-based box that comes with two network interfaces. It’s no surprise, that we got asked a lot if IPFire will support that box and so I wrote an email to them to ask if they can help us with that.
They didn’t even reply. They don’t even make the effort to talk to people, which want to help their (potential) customers to run interesting software on their hardware. Of course Utilite has been designed with router functionality in mind or for what other reason do you need multiple network adapters?
An other example: GlobalScale Technologies, who offer a new consumer box called MiraBox. It is very similar with Utilite.
We also did get no response from them – although IPFire already supports their DreamPlug, which was the first ARM-based device IPFire was running on.
Honestly, that’s okay with me, but it’s sad for all of you who wanted to see IPFire running on those devices. We cannot help you, if they don’t want to help us. Don’t get me wrong. I don’t want them to send me their hardware as a gift. I just need it for testing and I cannot do anything more than offering to work for free.
Option #2: Get the hardware support upstream
If they don’t want to support independent projects to help them, they could also do the entire work on their own. This basically means that the manufacturer, who designs new hardware, sends patches with the required changes to the Linux kernel maintainers.
If there was already working code for a piece of hardware in the Linux kernel and in the u-boot bootloader, which is the most commonly used bootloader on ARM, most of the work is already done, but I haven’t seen that anybody of the big manufacturers has ever done this. Not even the Raspberry Pi Foundation, which sold millions of their hardware bothered to do that. It was a group of hobbyists that rewrote lots of Broadcom’s code or wrote device drivers from scratch and got it merged into the Linux kernel.
The Raspberry Pi Foundation also did not reply to our emails which had patches in them to fix some things that were broken with their patches for the Linux kernel.
So here is what you need to do, dear hardware manufacturers: Provide working software with your hardware, because that will make it more valuable and demanded. You don’t need to do it yourself. As you can see, there are plenty of people around who want to help – for nothing in exchange. Let them help you.
At least reply! We are not mad at you if you don’t want any of our software running on your device, but replying to emails is the least you can do. An explanation about the reasons why you decline the offer would be brilliant, but not required. Not replying is just rude.
I can only speak for the IPFire project, when I say that we now don’t aim to support a wide range of ARM hardware any more and I think that lots of others fell the same way. We will be happy to add the low hanging fruits (enable support for boards that has already landed in the Linux kernel and u-boot) and that we can not and don’t want to help hardware manufacturers when they don’t help us. It’s just bad for you lot.
To the people who would like to run IPFire or any other distribution on ARM: Check if your distribution supports the hardware you want to use. It’s not a matter of course that if a distribution supports one or more ARM boards, that it will support them all.
They cannot be all that bad, you might think. No! There are people who got it right:
There are for example two projects which don’t only Open Source their code, but also the hardware design. Those two are Pandaboard and Wandboard, which both have public git repositories or patchsets that enable the Linux kernel to boot on this hardware.
Support for Pandaboard has not been available the day when the board was shipped for the first time, but it eventually landed in the Linux kernel and that’s great. A huge number of people worked on it to get this done and as far as I can see, Pandaboard is supported by all distributions that have an ARM version. I would like to see this for every hardware!
I am very sure that we are not going to see this – ever. Samsung is deploying a very similar mechanism to Intel’s SecureBoot, which basically requires to sign the code the system executes. For example, ODROID-X only boots with a signed bootloader that is signed by Samsung. They key for that is in the hardware.
Doing this has a lot of implications. Mainly that nobody can change the bootloader, because the system wouldn’t boot it when the signature is missing. Samsung could lose their private key and attackers would be able to use it to replace the bootloader by their own code which could do harmful things. There is no way to replace the key at any time – at least not a simple one.
The proprietary bootloader that is signed by Samsung will initialize the system and start an other bootloader that is also signed. That one then will run a version of u-boot that has also been signed. What you can see is a chain of signed code, that will only execute the next step if that is signed as well. But it all breaks with the signed version of u-boot, which is able to boot anything without any further validation! If I were an attacker, I would intercept the chain right here and replace the operating system kernel by my own one or add an additional layer of bootloaders… You can see that signing the code does not make any sense if you sign software, that does not check what it will boot next.
This is still not the only disadvantage. The provided version of u-boot is very old. It’s from 2010 and u-boot could be target of an attack. I have no chance to update it to a version that is fixed. If I had a problem with the hardware and need to fix a bug, I couldn’t. Source code of u-boot is available and I am sure the changes that Hardkernel (the company that sells the ODROID series) made are public somewhere, even if they are not merged upstream. But no matter what I would do with that, I could not run my own version of u-boot, because it does not have the right signature. It’s also impossible to run newer versions of u-boot that come with more features and improvements.
A mechanism that is supposed to protect me from malware now opens lots of chances for attackers to harm my system. If I am not able to protect myself against that and to fix my software, the system is not open. I don’t want to use it. Please stop developing this garbage!
Proprietary firmware blobs
So Samsung uses a chain of bootloaders to boot Linux on the EXYNOS family of System-on-Chips. What do these do? Nobody knows, except from what they tell us. The Raspberry Pi Computer comes with a proprietary firmware blob (= binary large object) as well, which is buggy so that IPFire won’t boot on systems with Hynix memory, but we cannot fix that. There is no source code available to these files and the vendors keep it a secret what is going on in there.
This harms innovation. The IPFire project has a policy to not ship such code in IPFire 3 any more, which means that we cannot support this hardware. It’s not open source and most of the time not free (i.e. legally prohibited), either.
Other distributions are doing exactly the same as the IPFire project does: The Fedora Project does not include support for the Raspberry Pi because of the proprietary firmware. Some other party made a so called Raspberry Pi Fedora remix. Debian has got a spin-off that is called Raspbian. You see where this is going? There is nothing Open Source about it.
If the board manufacturers want to see Open Source software running on their hardware, they need to get away from these binary firmware blobs. It’s possible to do that, because for example Pandaboard doesn’t have any. All of the code that is needed to boot this board is included in the official u-boot repository. It’s open and it’s free. The way it’s supposed to be.
The time and effort those people took to create remixes of major distributions could be well invested in proper research and development. Those remixes also claim that they are free – but not free as in free speech. They are only free as in free beer.
What is this all about?
tl;dr The ARM architecture as I see it has some serious problems. Not technical ones, but political ones. Vendors and manufactures choose to close the hardware down, so it is only able to run code they want. It is identical with Intel’s SecureBoot mess. Nobody wants this. There is no “Secure” in it. Stop it!
This is one of the two big reasons why there is poor support for ARM in IPFire and Linux distributions in general. If we want proper support for ARM hardware (and we have lots of proper reasons why we do want that), we cannot tolerate the decisions the vendors and manufacturers made for us.
@Manufacturers: I pointed it out several times, what we want you to do and what we want you not to do.
@Consumers: Don’t buy locked hardware and don’t buy hardware from people that don’t support free/open software. If you compare specifications in order to decide what hardware to buy, please take this into account. Write to the manufacturers about what you think is wrong with their products and that you would like to run your favourite distribution on it, but it is not supported. In the end, you are the people who define what products there are on the market. Demand for open hardware, boycott non-open hardware.
Sorry for the length of this post. I think that it was necessary to get this out, because the next time when we say that we won’t add support for some hardware to IPFire, you know why. As there are more and more suitable products that are perfect to run IPFire (just by their hardware specifications), we get more and more questions about porting IPFire to those products. Above is the long answer. This is the short one: We just can’t and we don’t want to.