Interview with the dev: The need for custom kernels

This week I am really excited about my interviewee, not only because I really love his work, but I do appreciate his frank approach to everything he does. I’ve used thalamus’ kernels on all of my Android devices and his kernels have improved the performance and battery life of my HTC Desire, Nexus S and now my Nexus Prime so I’m very familiar with his work and the benefits his kernels offer.

On occasion he can be brutal towards ignorant users, and it often results in me having a chuckle simply because I do find it funny when people who are using custom software on their handsets actually haven’t the faintest idea of what they’re doing. Rooting your phone and using those privileges to tweak your device accordingly should be a considered thing, I’ve done tons, literally hours and hours, worth of research to make sure that I understand exactly what it is I am doing, and I think more people who use custom Android software should do the same, but enough about me, I sent the developer some questions and these were his answers.

Gearburn: As a kernel developer, why do you see the necessity for custom kernels?

Thalamus: Firstly, unfortunately, some of the choices made with the stock kernels aren’t ideal. For example the default setting of CFQ is quite pointless for flash memory and is much more expensive in terms of CPU cycles because the code is massive compared to deadline. Another example of this is using the tree RCU for uniprocessor devices instead of the Tiny RCU. This choice appears to increase the number of scheduler ticks which fire, which causes more wakeups from idle states, and the performance and memory usage of tree RCU are worse. Also, stock kernels can have a number of debugging features enabled which aren’t needed. The performance gains from disabling such features are normally only measurable in micro benchmarks, but when you have a Cortex A8, you need all the help you can get…

Secondly, to focus on original development which allows me to work around existing issues or add new features. One example of this is my Nexus S IDLE2 project as well as my hotplug driver for the Nexus 7 which replaces the awful NVIDIA one. Another example is my work with cpuidle and extensive profiling of LP2 residencies for Nexus 7 and C state residencies for the Nexus Prime, which have resulted in additional power savings after I questioned them, profiled them and set them to saner values. – *GB – I can vouch for this as I’ve been helping Will test his Nexus Prime kernels.

The third reason I believe custom kernels to be necessary is updates. Stock kernels are rarely updated; meaning no bugfixes and no performance improvements. Being able to build your own means you can merge in the android-common development branches, or the android-omap development branches, which are development branches, so you have to be somewhat mindful of bugs. It also means you can cherry-pick specific fixes from mainline, or elsewhere. Or write your own fixes, as I mentioned above. A prerequisite of merging should be an understanding of what exactly is being changed, however. Cherry-picking random stuff from all over the internet to have a large feature list is not cool and the result will be a mess full of subtle issues. And Yes, I did have to mention that :p

GB: What is your development ethos?

Thalamus: My development ethos is to keep everything as simple as possible, within reason. Why use a driver which is 2000 lines of code when an alternative of 200 lines works as well, or even better? Obviously, if the 2000 line driver provides significant performance advantages, you would use that. But generally, smaller is better, especially for embedded devices.

I also investigate possible implications of whatever change I am planning or whatever code I am merging. Most kernel compilers / cherry-pickers don’t do this, and end up with subtle or serious issues as a result. Obviously you can’t understand every line of code in the kernel, but you can generally have an inkling by looking at the patch whether it is likely to cause an issue or not, although, subtle issues caused by random things are a kernel favourite.

I won’t add features to follow the crowd. My kernel is my project. I add the features which I deem add value to it. Adding 100 features to inflate the feature list doesn’t add value, it just makes my kernel exactly the same as all the others and if you go back to the answer about how I got into kernel developing, you will see that avoiding the crap was my main reason for getting into it. This is why I refuse feature addition requests 99% of the time. I think being strict with that has kept my kernel stable, lean and not the product of pointless merging of code like…*cough* Air Kernel.

I am deliberate and my changes/improvements are thoroughly thought out. If I’m not happy, the patch doesn’t go in. If it doesn’t make any difference, the patch is removed. I regularly review my commits and rebase out stuff I deem unnecessary.

I also aim to fix problems and improve the stock kernel. My aim is not a fork of stock with a million features and patches added. My aim is stock plus fixes and enhancements which add value. If those fixes add up to 1500 lines of code, like IDLE2, so be it. In my opinion, the functionality which I have implemented should already have existed. It exists in the OMAP4 kernel and it exists in the Tegra 3 kernel.

And finally, to chastise morons and lazy people on a regular basis. If you are a moron please don’t talk to me because you won’t like the response and please stop posting on my threads, that way you will not be ridiculed. I don’t chastise people who take the time to read threads or attempt to help themselves before expecting me to help them.

GB: If you could summarise your kernels in a single caption, what would it be?

Thalamus: Individual, lean and stable with no useless or pointless features or ‘backports’.

GB: What is your current daily device?

Thalamus: It’s currently a toss-up between the Lumia 800 and the Nexus Prime. I love the Lumia for its utter simplicity, smoothness and ease of use. I love the Prime for the amazing screen and Google Play Music because I can access my entire music collection from there.

GB: Which is your favourite Android device up to now?

Thalamus: Nexus 7 or Nexus Prime. They are both pretty amazing devices. Android without manufacturer UX ‘enhancements’ is so much nicer to use, plus you get the newer versions first. I will never again recommend a non-Nexus Android device.

GB: How did you get into kernel developing?

Thalamus: That’s an interesting question. I got into kernel developing because I realised that none of the available ones on XDA (the biggest Android development platform) were suitable for what I wanted. I tried various kernels back on my Bravo (HTC Desire), but most of them were utter garbage, full of pointless ‘features’ and bloat, unstable undervolting and overclocking and basically they were just crap. Even the stable ones, like the CyanogenMod kernel was as slow as hell because of bad choices (see my comments above about bad choices).

So, I decided to figure out how to build them and make my own. Most of my early attempts weren’t great, but after a bit of perseverance, I managed to get something reasonably stable. I must point out that none of what I was doing was developing; it was adding a few patches and compiling a kernel.

Developing means that you are improving or ‘developing’ the product, not just cherry-picking other people’s work. Alas, the abuse of the word ‘developer’ on XDA has made it meaningless, but I digress.

So, from there, after a while a few people started asking to try my kernels. Eventually, it made its way into Oxygen ROM by AdamG, but at that point I still hadn’t really released standalone kernels and I still wasn’t really developing because I wasn’t writing any original code. Anyway, to cut a long story short, my actual developing started and I became a real kernel developer when I got a Pyramid (HTC Sensation). I wrote a hotplug driver for that, and although it was awful and full of ugliness, it worked better than Qualcomm’s stock alternative. I never did release that kernel, because I really wasn’t bothered about the fame or fortune of it all, I did it for the sake of learning and to solve a problem.

My main motivation for becoming a kernel compiler / cherry-picker (to a small extent) was the crap offerings that existed. My main motivation for becoming a kernel developer was to solve a problem with my own ideas/solutions.

GB: Is there a specific reason why you don’t develop ROMs as well? If so, why?

Thalamus: Since ICS, ROM development simply doesn’t interest me. I simply don’t see the point. Stock is stable, it’s as fast or faster than any custom ROM and it has all the features any normal person needs so custom ROMs are pretty much superfluous. It also provides a stable base for my kernel testing, because what you don’t want is to have a bug and spend ages chasing it in the kernel only to find that it is caused by some crappy custom ROM.

GB: What do you think of the improvements that have been made to the OS over the past two years, because I have used Android since 2.2 on my HTC Desire, then I had a Nexus S and now I’ve got a Nexus Prime, I think that Android has got to a point where what Google is releasing is actually so polished that we don’t NEED custom ROMs to try improve our devices anymore, it’s more for fun, would you call that a fair comment?

Thalamus: I think it is much better. I actually abandoned Android for WP7 after Gingerbread. I hated it. It was slow, buggy, laggy, with jerky animations, you name it, it sucked. Ice Cream Sandwich (Android 4.0) was a good stepping stone, but that still wasn’t too great, especially on the Nexus S. JellyBean, however, is pretty damn good. It’s not perfect, and it’s not WP7 smooth yet, but it’s not a million miles away and it’s light years closer than Gingerbread was. I agree that custom ROMs aren’t needed and I think the majority of them are ‘developed’ for the wrong reasons, ie, ego and fame.

GB: What do you make of all the current patent lawsuits involving Android and iOS? Do you think it’s hampering the development of these two incredibly popular technologies?

Thalamus: It’s quite comical and pathetic really. Alas, I think Samsung opened the can of worms by copying the look and feel and user interface of the iPhone 3G with its Galaxy S device which was a top hit for Samsung. I still find it mildly amusing when Android fanboys try to deny that.

I’m not sure about it hampering development, as development will go on regardless. You could actually argue that it may help development in the long-term, because it will force software engineers to create original ideas for functionality instead of using patented ideas, which may be better than existing and commonly used functionality. Take WP7 as a good example of that, the user experience is completely unique compared to Android and iOS.

GB: Which 3 Android apps could you not live without?

Thalamus:

• Play Music so I can access and stream my entire 8500 track music collection.

• System Panel is useful for finding causes of wakeups, battery drain and is also handy if you just want to kill everything that is running.

• Gmail is easily the best email service and the app has always been one of my favourites.

GB: What would you like to see in the next Nexus device?

Thalamus: OMAP5 – I really don’t want to have to learn yet another proprietary SoC (system on a chip), specially a Samsung one. TI (Texus Instruments) makes the best SoCs anyway, and their kernel code tends to be high quality and often is somewhat elegant. Their SoCs are also open, there is documentation and help freely available to developers, unlike Qualcomm, Samsung, Nvidia, et al. By the way, does anyone have a tegra3 TRM I can have, completely confidentially of course?

Other features I’d love to see include a 4.65″ Super Amoled HD display (preferably no pentile), a decent Camera as the one on the Nexus Prime is utterly rubbish, a metal body, sick of the Samsung cheap plastic crap, but other than that, an updated Nexus Prime would be fine by me.