Ancient MIPS Cross Compilation

06 Aug 2022

For my IoT security research project at UMD’s Breakerspace, I recently needed to compile a custom binary to run on smart devices for experimentation. Previous experimental setups (not designed by me) used a series of Bash scripts, but I discovered they were barely reaching the device max capacities.

Many IoT devices run embedded architectures like ARM or MIPS(EL), not x86. So I needed to cross-compile my C code to run on the right architectures. I thought I was clever for managing to install gcc-<arch>-linux-gnu on Ubuntu.

Until I tested my code on one particular MIPS device, which gave me a FATAL: kernel too old. What?

uname -r returned the kernel version as 2.6.36+. This kernel version was released in 2010 - talk about an ancient kernel! For reference, the current one is like 5.19. So, I set about trying to compile to target this kernel version for MIPS.

4 days later and 8+ builds of binutils / GCC / gLibc later, I decided nobody else should ever go through the same headaches I went through to compile just a hundred lines of C.

Cross Compilation

Turns out I wasn’t as familiar with how Linux systems worked as I thought. In order to cross compile, I needed to compile binutils (ld, etc) for MIPS first. Then, I needed to compile GCC for MIPS, but not really because GCC needs gLibc. But to compile gLibc for MIPS, I needed a compiler for MIPS. Which is GCC! What the hell!!

After scouring many a StackExchange site, I found crosstool-ng. This tool promised an end to my troubles. But that is where they really began.

First, I tried to pass in --enable-kernel=2.6.36 as a compiler flag. But that didn’t work. Then I found out recent versions of glibc dropped support for older kernel versions. Then I ran into a bunch of issues with incompatible binutils / gcc / gLibc versions. Then I thought I fixed my issues, but it turns out I really just forgot to re-configure crosstool to target MIPS and instead targeted a different architecture.

But, nobody wants to know about that. You are just here to learn to compile ancient MIPS binaries.

First, clone, build, and install crosstool-ng from its GitHub following the online instructions.

Then, run ct-ng menuconfig and change the following options.

• Enable “Use obsolete features”
• Set target architecture to “MIPS”
• Set endianness correctly
  • MIPS is big endian
  • MIPSEL is little endian
• Set target OS to Linux
• Set binutils version to 2.26.1 (lowest)
• Set gLibc version to 2.23
• Set GCC version to 6.5.0

Now, run ct-ng build. Wait for a couple hours for the build to finish. Come back to an error in the final stages of the build.

The issue seems to have been reported before, with a fix for it being merged in this PR.

So, pull from GitHub again, rebuild ct-ng, reconfigure the build, restart the build. And run into the same error!!

For some unholy reason the patch file just does not get applied. Maybe I’ll make a PR for it when I get less busy with work. To fix this for now, just manually apply the patch file to sysdeps/unix/sysv/linux/mips/vfork.S in the gLibc source build directory, usually in ~/.build/.

Then, rebuilding should give you a nice and shiny MIPS toolchain for kernel version 2.6.36! You can proceed by adding the bath containing these binaries to your system PATH variable.

Bonus round: OpenSSL!

My code also required hash implementations from OpenSSL. This process is a bit easier and less frustrating.

First, clone the repository and checkout the version branch you want.

Then configure your OpenSSL build by running:

./Configure --prefix=/usr/mipsel-unknown-linux-gnu linux-generic32

and build and install by running:

make -j4 CC=mipsel-unknown-linux-gnu-gcc; sudo make install

Because it sort of depends on where your mipsel-unknown-linux-gnu-gcc toolchain lives, you will have to manually install the .so and .a files. You can just enable verbose output when linking to find the errors (-Wl,--verbose). Then just find the failed include paths and then add the compiled libraries there.

🥳!