[kelnos]

Yes, this was my first thought as I was reading the article. A better title for it would be "Backporting Modern Linux Kernel Features to Our Really Old Kernels Instead of Doing the Right Thing and Keeping Up To Date".

I do -- I really do -- appreciate that version churn is difficult, and the Linux kernel also doesn't make it easy since they don't guarantee any stable internal APIs, but they're also adding a lot of work on themselves by having to maintain their own kernel trees that diverge significantly from mainline. They're also at the mercy (to some extent) of many chipset manufacturers and whatever they've chosen to base their efforts on.

A quick look at some kernel release timeframes from the versions they mention in the article:

    4.3  - 11/2015
    4.4  - 02/2016
    4.6  - 05/2016
    4.8  - 10/2016
    4.10 - 02/2017

The only kernel out of that list I'd unquestioningly accept as it being unrealistic to upgrade to for Oreo is 4.10. 4.8 might be a stretch since I'm guessing they'd already branched internally for Oreo by then, though they likely had a month or so of RCs that they could have used as a base before that. There's certainly risk to basing your work on a newly-released (or soon-to-be-released) kernel, but given the general high quality of kernel releases, I imagine that'd be pretty far down on their list of risks. Regardless, 4.6 or 4.7 would be entirely reasonable to use as a base, and since they own the conformance test criteria, they could also require that all their vendors use that as a minimum version.

And yet, they are backporting some features as far back as 3.18, which was originally released in December of 2014, and, while it was designated a LTS kernel, it, at this point in time, has moved into end-of-life status. And we wonder why Android security is a nightmare.

[c2h5oh]

Main reason is that there is a lot of hardware that never got binary blobs / drivers updated for newer kernels.

We're talking input controllers, wifi, bluetooth, nfc chips, gyroscopes, amps and half a dozen other parts that never attempted to have drivers mainlined in Linux kernel.

It's a chicken and egg problem - they won't make updated blobs until Android doesn't include newer kernel. Android won't use newer kernel because that would block 2/3 of the phones from being upgraded because of Broadcom bluetooth chips alone - Samsung might decide they are better off forking Android than waiting for Broadcom.

Edit: Sony has a bunch of phones that can run on 4.4 kernel: https://developer.sonymobile.com/open-devices/

[kelnos]

Yes, and I addressed that in my comment. Google, however, can effectively do whatever they want. If they require kernel X, and a manufacturer doesn't support it, they'll either get their shit together, or they'll get left behind. I bet most of them do enough business supplying parts for Android phones that they'd get their shit together. And it' not hard! Writing an initial driver for some hardware might take a lot of effort, but keeping it up-to-date as new kernel releases happen will not, at least in the vast majority of cases.

[zaphirplane]

I expect if google had that strength in position, they would. Perhaps that would lead to more fragmentation and less vendors releasing the latest android, leading to a highly dominant maker squashing the others and then having a stronger position in negotiating with google.

[thanksgiving]

Perhaps it would be more desirable to reduce dependency on blobs? What can we do to encourage manufacturers to release source for their hardware? I assume they care about selling hardware and the firmware is just incidental?

[dmm]

Even when full source is available it doesn't really solve the problem. Many of the drivers provided for android socs are very poor quality and would not be allowed in the kernel. Typical problems include not using linux conventions for config parameters(device tree) and duplicating large portions of existing kernel functionality.

Not getting into the tree is a problem because kernel interfaces change all the time. When someone changes a kernel interface they are expected to update all of the affected code, but out-of-tree drivers don't get that.

[_pmf_]

> If they require kernel X, and a manufacturer doesn't support it, they'll either get their shit together, or they'll get left behind.

This leaves Google with a version of Android that does not run on anything. There are less than a handful of relevant SoM manufacturers that are capable of delivering consumer grade SoMs capable of running hardware accelerated Android; Google can not alienate these.

[kelnos]

Apple seems to have no problem keeping drivers/blobs for their hardware working when they release new versions of iOS. Sure, they do have the advantage of tight control over their hardware and core software, and a vastly smaller number of pieces of hardware to target, but in that way they're not that much different than any random Android vendor, hardware-wise. Sony (for example) is perfectly capable of only choosing vendors that can keep up with kernel versions, or at least vendors that will be open enough with them (not even with the public, just Sony) so that Sony can hire a software team to keep things up to date.

But they don't care enough about this sort of thing (unlike Apple), and no one (such as Google) is forcing them, so it won't get done unless they see an economic upside.

[pjmlp]

For sure Google can do it, what would they do, sell handsets with their own OS, based on a fork from GNU/Linux?

It has worked quite well for those that tried.

[majewsky]

What would they do? Sell handsets with a years-old Android, of course. Experience shows that the average customer doesn't give a shit about Android versions.

[pjmlp]

Which in such scenario wouldn't be able to talk to Google Play Service servers any longer, if Google was actually serious about doing it.

[yahna]

It also leaves those manufacturers without anything to put on their hardware.

I would think they would start to take things more seriously at that point.

[_pmf_]

I'm insure about who has the upper hand, but I feel like SoM manufacturers know what they are doing and are where they are based on merit whereas Android is there because it was available when it mattered and gained momentum, not because of any technical merit. Android as a developer ecosystem is a train wreck. I have better tooling for deeply embedded bare metal platforms than I have for Android userspace applications.

[tbrock]

It's not like more than 1% of android phones that have shipped to date will actually get an android update to Oreo anyway so why not leave them behind and update the kernel?

New phones need new drivers which must support the new kernel, period. I don't see what the big deal is. You aren't getting Oreo on your old ass Samsung Galaxy S2 anyway.

[petecox]

Nougat runs fine on a Galaxy S2 with Lineage OS. I would not be unexpected for them to port Oreo eventually too.

[saghm]

GP's point was about OEM support for newer versions of Android, though; I can't imagine that third-party ROMs on old devices are even slightly a factor in Google's hesitation to upgrade to newer kernel versions

[xbmcuser]

Most manufactures don't update their phones to newer version of Android anyway. I think Google should bite the bullet and upgrade to the newer kernal or atleast inform now that the next year Android will be on the latest kernal so hardware manfacturers should upstreaming their patches.

[endorphone]

The Galaxy S8 runs 4.4, and has since release. Oreo the AOSP uses the 4.10 headers. But ultimately any vendor can integrate with a wide variance of kernels depending upon their needs.

[structural]

If they are backporting features well, they're approaching kernel development very similarly to the datacenter.

As an example, the standard server at $large-dayjob-server-farm runs on RHEL 6, with RHEL 7 (the latest release) being used for new machines. Both are still supported for quite a few years into the future. Let's look at their kernel versions: (per https://en.wikipedia.org/wiki/Red_Hat_Enterprise_Linux)

"6.9, also termed Update 9, March 21, 2017; 5 months ago (kernel 2.6.32-696)"

"7.4, also termed Update 4, August 1, 2017; 28 days ago (kernel 3.10.0-693)"

Upstream release timelines: 2.6.32 - Dec 2009 3.10.0 - Jun 2013

Both of these upstream releases are YEARS older than both "current" and what many Android devices are using today - we'll have machines running some patched version of 2.6.32 a decade after its initial release!

So there's obviously a difference in quality of backports and updates here and some grey area in between. I can't with a straight face say that either Red Hat or Google are right or wrong in their approaches here, it's just very different than the frequent-release model that some people are used to.

[kelnos]

That's a very good point.

My understanding, though, is that RH etc. do that mainly for stability reasons. Their customers don't want latest-and-greatest, they want small evolutions of the stuff they know works, with only bug-fixes and must-have new features.

Of course, that comes with downsides too: build your infra on RHEL 6.x, and then once 6.x becomes end-of-life, upgrading to the new latest release is a huge undertaking.

I think RHEL is doing the right thing because it's what their customers want; I would argue that in many cases their customers are doing the wrong thing and should make more-frequent, smaller upgrades rather than one giant upgrade twice a decade. But that's certainly open for debate and reflects only my experience managing infra ;)

With something like Android, I'd expect the pendulum to swing a bit the other way: you don't want bleeding-edge or unstable, but you probably do want something quite a bit more up-to-date than something that RH pushes out. To use your example, I'd say releasing a phone in 2017 based on 2.6.32 (released nearly 8 years ago!), even with 696 patch releases, would be unacceptable.

But yeah, where do you draw the line at acceptability? I personally think you shouldn't be going with a kernel release whose series was released more than a year or so before you start development (assuming getting to a release is going to take 6-12 months from that point), and ideally you just take whatever is the latest stable when you start development, if that's possible. Sure, opinions differ, but that's kinda the point... this is my opinion :)

[zaphirplane]

The comparison isn't fair you as a user of RH, expend an effort to upgrade and the benefits are often not clear so you don't

RH as a vendor has to backport because their users demand a long supported and modern-ish kernel

Google spend time and effort to backport and this story is discussing the benefits vs putting that effort in upgrading.

Your analogy isn't fair, thou obviously android's customers here are the device makers and they are tilting the balance towards backport