New Build System

• December 20, 2015

All professional software companies automate their builds and test as much as possible automatically as well with the intention being to ensure consistent quality over several releases. There is a huge industry providing software and services for automated tests and releases. You find huge amounts of articles outlining why you should automate, which tools to use and telling you if you do not do this you are doing things wrong. What they fail to talk about is the work needed to get to the point where your build pipeline is actually working.

As a micro-company, build automation became really important after crossing the 100K download barrier and each update could potentially disrupt more people than living in the town we are in. In addition, with a growing software comes additional burden regarding tests. A two page software is easily clicked through in a few minutes. With over 30 pages things look a bit different, so the first target was to at least have a test opening each page making sure the app does not crash. Checking on the app behavior for each page was next.

The tools of choice are, of course, Jenkins and the Android Emulator Plugin. The starting point was a virtual machine provided by Turnkey Linux running in VirtualBox on a Mac Mini. At that point in time we were still using Mercurial as the source code provider, also hosted on the Mac Mini in another virtual machine provide by Turnkey Linux.

After some minor problems with the Jenkins appliance which was already discussed and resolved on the Turnkey Linux forums, things were up and running in about half a day of work. This does not mean there was a working build pipeline, just a working Jenkins installation. Setting up the jobs to get the sources and building release versions was another day of work. Each step itself sounds easy to do but they tend to fail at least once as my setup never exactly matched the setup of the descriptions online. Problems aside, this paid for itself after about 10-20 releases as release builds tend to completely block the development machine and there are several steps involved which can go wrong if done manually. Getting the tests running in an emulator as part of a Jenkins build turned out to be an impossible problem to solve for me. I write fairly complicated software for a living but mostly on Windows so Linux needed some rethinking and lots of manuals. Trying to get an Android virtual machine running inside an Debian virtual machine with only the command shell and the log output in the Jenkins job is no fun and took up a few days of work without any results.

Clearly this approach did not work and needed some rethinking. If running the Android emulator within the VM did not work, why not run the whole thing directly on the Mac Mini? All we need to do is install Jenkins, the Android stuff is already there as the Android Studion is installed. And the Emulator starts from within Android Studio, what could go wrong?

In the beginning, not much did go wrong except the installation package for Jenkins alone did not lead to a working solution. The package was installed but the server did not start up. After poking around the internet for half a day somebody mentioned Homebrew and with its help getting a working Jenkins installation turned out to be fairly easy. Since we already had jobs for building the app and theoretically running tests in the VM, getting them to work in the new Jenkins installation was not big deal. At this point in time we were using Git as a source code repository and Nexus OSS inside an Ubuntu VM as the repository manager for released libraries. Setting up Nexus used up about a day of work as well but also paid off by having more control over which packages are used where and sharing them across our three apps. Git took only a few minutes thanks to the guys at Synology, it took a bit longer figuring out the permissions and integrating the public keys of the build machines to access the repositories via SSH. The Git repository was on a separate machine but the Nexus installation needed to run inside a VM so VirtualBox was a requirement. So far, so good. Unfortunately, it turns out you can not run both the Android Emulator and VirtualBox at the same time on OS X.

Most people would despair at this point and I was not an exception. Since VirtualBox and the Android emulator could not live together they had to be separated. Another Mac was out of question due to price and convenience and a Windows machine not necessary a requirement, Ubuntu would do as well. The system of choice was an Intel NUC due to the reasonable price and the option to add your own ram and SSD hard drive. Being small and practically noiseless was also a factor. The system was easily put together, just unscrew the bottom insert the ram and the SSD, screw the bottom back on and you are done. The Ubuntu installation process was painless and Jenkins was up and running in about half a day mostly waiting for updates. Getting the Android build process up and running took another day figuring out access rights and stumbling over the 32bit SD card creation tool in a 64bit system problem. Nothing unsolvable, it just took time. In the end the Ubuntu system turned out to be more convenient than the OS X simply because both copy / paste and each key on my keyboard worked over VNC from Windows. This may sound trivial but if you can not enter a dash character unless using the virtual keyboard you tend to be slightly annoyed.

The Android emulator also provided some challenges but it actually ended up working after specifically switching off showing the emulator window. With the build pipeline in place the only problem left was the need to switch on three machines and make sure the VM is started up in the Mac. Since the only thing left on the Mac was the Nexus installation and we did have space left on the NUC trying to move things to the new machine sounded like a good idea. This turned out to be the most pleasant surprise of the whole process. Moving Nexus from one installation to another involves simply copying / replacing the data folder. The complete repository plus all users are included and things just work.

In the end, it would have been best to go with a NUC and use a dedicated machine instead of VMs from the beginning. It would have worked getting everything running on the Mac as well but failed due to configuration issues. In the end, build automation is still difficult. Everybody likes to show nice graphs of the steps supposed to happen after each other but rarely is the amount of work and knowledge mentioned you need to get to this point. Things are better than a few years ago but there is still room simplifying the setup process of a build pipeline. You can definitely get things working but just sorting through the options and deciding which way to go and then getting there can be more work than you think.