[mrhigat4]

Android should really use a modern kernel. All the forking mess involved in Android updates is a terrible problem predicated by the lack of generic drivers on mobile devices.

Copperhead[0] has been working to apply security patches to the kernel for some time and PostMarketOS[1] has an eventual goal of using the mainline upstream kernel. Really pulling for PMOS.

[0]: https://copperhead.co/android/

[1]: https://www.postmarketos.org/

[kelnos]

Yes, this was my first thought as I was reading the article. A better title for it would be "Backporting Modern Linux Kernel Features to Our Really Old Kernels Instead of Doing the Right Thing and Keeping Up To Date".

I do -- I really do -- appreciate that version churn is difficult, and the Linux kernel also doesn't make it easy since they don't guarantee any stable internal APIs, but they're also adding a lot of work on themselves by having to maintain their own kernel trees that diverge significantly from mainline. They're also at the mercy (to some extent) of many chipset manufacturers and whatever they've chosen to base their efforts on.

A quick look at some kernel release timeframes from the versions they mention in the article:

    4.3  - 11/2015
    4.4  - 02/2016
    4.6  - 05/2016
    4.8  - 10/2016
    4.10 - 02/2017

The only kernel out of that list I'd unquestioningly accept as it being unrealistic to upgrade to for Oreo is 4.10. 4.8 might be a stretch since I'm guessing they'd already branched internally for Oreo by then, though they likely had a month or so of RCs that they could have used as a base before that. There's certainly risk to basing your work on a newly-released (or soon-to-be-released) kernel, but given the general high quality of kernel releases, I imagine that'd be pretty far down on their list of risks. Regardless, 4.6 or 4.7 would be entirely reasonable to use as a base, and since they own the conformance test criteria, they could also require that all their vendors use that as a minimum version.

And yet, they are backporting some features as far back as 3.18, which was originally released in December of 2014, and, while it was designated a LTS kernel, it, at this point in time, has moved into end-of-life status. And we wonder why Android security is a nightmare.

[c2h5oh]

Main reason is that there is a lot of hardware that never got binary blobs / drivers updated for newer kernels.

We're talking input controllers, wifi, bluetooth, nfc chips, gyroscopes, amps and half a dozen other parts that never attempted to have drivers mainlined in Linux kernel.

It's a chicken and egg problem - they won't make updated blobs until Android doesn't include newer kernel. Android won't use newer kernel because that would block 2/3 of the phones from being upgraded because of Broadcom bluetooth chips alone - Samsung might decide they are better off forking Android than waiting for Broadcom.

Edit: Sony has a bunch of phones that can run on 4.4 kernel: https://developer.sonymobile.com/open-devices/

[kelnos]

Yes, and I addressed that in my comment. Google, however, can effectively do whatever they want. If they require kernel X, and a manufacturer doesn't support it, they'll either get their shit together, or they'll get left behind. I bet most of them do enough business supplying parts for Android phones that they'd get their shit together. And it' not hard! Writing an initial driver for some hardware might take a lot of effort, but keeping it up-to-date as new kernel releases happen will not, at least in the vast majority of cases.

[zaphirplane]

I expect if google had that strength in position, they would. Perhaps that would lead to more fragmentation and less vendors releasing the latest android, leading to a highly dominant maker squashing the others and then having a stronger position in negotiating with google.

[thanksgiving]

Perhaps it would be more desirable to reduce dependency on blobs? What can we do to encourage manufacturers to release source for their hardware? I assume they care about selling hardware and the firmware is just incidental?

[dmm]

Even when full source is available it doesn't really solve the problem. Many of the drivers provided for android socs are very poor quality and would not be allowed in the kernel. Typical problems include not using linux conventions for config parameters(device tree) and duplicating large portions of existing kernel functionality.

Not getting into the tree is a problem because kernel interfaces change all the time. When someone changes a kernel interface they are expected to update all of the affected code, but out-of-tree drivers don't get that.

[_pmf_]

> If they require kernel X, and a manufacturer doesn't support it, they'll either get their shit together, or they'll get left behind.

This leaves Google with a version of Android that does not run on anything. There are less than a handful of relevant SoM manufacturers that are capable of delivering consumer grade SoMs capable of running hardware accelerated Android; Google can not alienate these.

[kelnos]

Apple seems to have no problem keeping drivers/blobs for their hardware working when they release new versions of iOS. Sure, they do have the advantage of tight control over their hardware and core software, and a vastly smaller number of pieces of hardware to target, but in that way they're not that much different than any random Android vendor, hardware-wise. Sony (for example) is perfectly capable of only choosing vendors that can keep up with kernel versions, or at least vendors that will be open enough with them (not even with the public, just Sony) so that Sony can hire a software team to keep things up to date.

But they don't care enough about this sort of thing (unlike Apple), and no one (such as Google) is forcing them, so it won't get done unless they see an economic upside.

[pjmlp]

For sure Google can do it, what would they do, sell handsets with their own OS, based on a fork from GNU/Linux?

It has worked quite well for those that tried.

[majewsky]

What would they do? Sell handsets with a years-old Android, of course. Experience shows that the average customer doesn't give a shit about Android versions.

[yahna]

It also leaves those manufacturers without anything to put on their hardware.

I would think they would start to take things more seriously at that point.

[_pmf_]

I'm insure about who has the upper hand, but I feel like SoM manufacturers know what they are doing and are where they are based on merit whereas Android is there because it was available when it mattered and gained momentum, not because of any technical merit. Android as a developer ecosystem is a train wreck. I have better tooling for deeply embedded bare metal platforms than I have for Android userspace applications.

[tbrock]

It's not like more than 1% of android phones that have shipped to date will actually get an android update to Oreo anyway so why not leave them behind and update the kernel?

New phones need new drivers which must support the new kernel, period. I don't see what the big deal is. You aren't getting Oreo on your old ass Samsung Galaxy S2 anyway.

[petecox]

Nougat runs fine on a Galaxy S2 with Lineage OS. I would not be unexpected for them to port Oreo eventually too.

[saghm]

GP's point was about OEM support for newer versions of Android, though; I can't imagine that third-party ROMs on old devices are even slightly a factor in Google's hesitation to upgrade to newer kernel versions

[xbmcuser]

Most manufactures don't update their phones to newer version of Android anyway. I think Google should bite the bullet and upgrade to the newer kernal or atleast inform now that the next year Android will be on the latest kernal so hardware manfacturers should upstreaming their patches.

[endorphone]

The Galaxy S8 runs 4.4, and has since release. Oreo the AOSP uses the 4.10 headers. But ultimately any vendor can integrate with a wide variance of kernels depending upon their needs.

[structural]

If they are backporting features well, they're approaching kernel development very similarly to the datacenter.

As an example, the standard server at $large-dayjob-server-farm runs on RHEL 6, with RHEL 7 (the latest release) being used for new machines. Both are still supported for quite a few years into the future. Let's look at their kernel versions: (per https://en.wikipedia.org/wiki/Red_Hat_Enterprise_Linux)

"6.9, also termed Update 9, March 21, 2017; 5 months ago (kernel 2.6.32-696)"

"7.4, also termed Update 4, August 1, 2017; 28 days ago (kernel 3.10.0-693)"

Upstream release timelines: 2.6.32 - Dec 2009 3.10.0 - Jun 2013

Both of these upstream releases are YEARS older than both "current" and what many Android devices are using today - we'll have machines running some patched version of 2.6.32 a decade after its initial release!

So there's obviously a difference in quality of backports and updates here and some grey area in between. I can't with a straight face say that either Red Hat or Google are right or wrong in their approaches here, it's just very different than the frequent-release model that some people are used to.

[kelnos]

That's a very good point.

My understanding, though, is that RH etc. do that mainly for stability reasons. Their customers don't want latest-and-greatest, they want small evolutions of the stuff they know works, with only bug-fixes and must-have new features.

Of course, that comes with downsides too: build your infra on RHEL 6.x, and then once 6.x becomes end-of-life, upgrading to the new latest release is a huge undertaking.

I think RHEL is doing the right thing because it's what their customers want; I would argue that in many cases their customers are doing the wrong thing and should make more-frequent, smaller upgrades rather than one giant upgrade twice a decade. But that's certainly open for debate and reflects only my experience managing infra ;)

With something like Android, I'd expect the pendulum to swing a bit the other way: you don't want bleeding-edge or unstable, but you probably do want something quite a bit more up-to-date than something that RH pushes out. To use your example, I'd say releasing a phone in 2017 based on 2.6.32 (released nearly 8 years ago!), even with 696 patch releases, would be unacceptable.

But yeah, where do you draw the line at acceptability? I personally think you shouldn't be going with a kernel release whose series was released more than a year or so before you start development (assuming getting to a release is going to take 6-12 months from that point), and ideally you just take whatever is the latest stable when you start development, if that's possible. Sure, opinions differ, but that's kinda the point... this is my opinion :)

[zaphirplane]

The comparison isn't fair you as a user of RH, expend an effort to upgrade and the benefits are often not clear so you don't

RH as a vendor has to backport because their users demand a long supported and modern-ish kernel

Google spend time and effort to backport and this story is discussing the benefits vs putting that effort in upgrading.

Your analogy isn't fair, thou obviously android's customers here are the device makers and they are tilting the balance towards backport

[tadfisher]

Agreed, but let's point fingers at the whole stack if we want this to happen.

Treble [0] will finally add a HAL to the base system, so updates to the kernel and drivers should in theory be easier on devices that ship with 8.0 (of which there are exactly zero so far). So at least Google's on the right track.

But Google doesn't create the drivers and Google doesn't ship the board support packages which vendors build upon. And so we come to Qualcomm, Samsung, Mediatek, and whoever else is shipping proprietary drivers for their SoCs and radios, and who don't provide binaries for new ABIs.

The workaround is libhybris, which Ubuntu Touch, Sailfish, and now PMOS are using to support drivers built against Android kernels and its userspace, but that is so fraught with issues that it can't possibly be supported by Google or the vendors.

So we have Android 8.0 shipping with a 4.4 minimum to support the lowest common denominator of BSPs, and it kind of has to be that way until some market force changes the landscape. I don't have high hopes for either of the projects you mentioned, sadly.

[0] https://source.android.com/devices/architecture/treble

[richdougherty]

Agreed, it really is a shared problem. It seems like a kind of "collective action problem" [1], where coordination between different parties is required - but they all have different interests and they're not all interested in splitting the costs.

Google is in a difficult position. In order for Android to catch up to the iPhone they needed to encourage a wide consortium of vendors to adopt and sell Android hardware. Part of the way they encourage vendors to sell Android phones is by being very permissive about how vendors install, update and use Android. This helps Android by growing market share.

Unfortunately, it leads to a lot of variation and fragmentation in devices, which in the long term can hurt the Android ecosystem. I guess there's a balance that Google is trying to strike between being too permissive and too restrictive in how they work with vendors.

[1] https://en.wikipedia.org/wiki/Collective_action#Collective_a...

[pjmlp]

It worked for Microsoft to impose hardware designs on PCs, the big difference was that OEMs did not had the source code of MS-DOS and later Windows available to them, to do whatever they felt like it.

[thanksgiving]

OEMs can't do whatever they like with Android outside of China PR. They need Google play services.

[pjmlp]

Yet Google decides, sadly for us, not to use that as enforcement for the updates.

[mtgx]

Indeed. Google has the power, and that excuse that Android is open source and therefore Google can't impose anything on OEMs is getting a little old and tired.

If Google would have promoted Android the way it did with Chrome OS (and the open source Chromium OS), it wouldn't be in this situation, and we'd be getting updates often. But I guess hindsight is 20/20. I still wish they did more about the support of Android devices throughout the ecosystem, not just for the highest-end devices.

[petecox]

> we come to Qualcomm, Samsung, Mediatek, and whoever else is shipping proprietary drivers for their SoCs and radios, and who don't provide binaries for new ABIs.

Vote with your wallet.

The librem 5 mentioned this week uses a 'liberated' i.MX6 chip because they want a phone with an upstream kernel. Technology wise, it's a little long in the tooth, but they're emphasising it's possible to not reward vendors for bad behaviour.

With RPi providing resources to de-blob and upstream the Pi, perhaps some entrepreneur will get behind a crowdfunded Broadcom based phone, with work underway on VC5.

http://phoronix.com/scan.php?page=news_item&px=BCM7268-DRM-W...

[exDM69]

Yes, they should use modern kernels but that has not happened yet. I am appalled by the practice of keeping customers on old software and sometimes even without proper security patches.

I hope regulators (EU in particular) will act on this and require manufacturers to provide support for products for a reasonable amount of time.

Devices should be clearly labelled with a "best before" date, until which the vendors should provide (at least) security updates. Right now the situation is unbearable, you buy a phone for $200 to $700 and you don't know how long it is good for. A technically oriented or security minded person can deal with the situation but almost everyone has a smartphone. Billions of unsecure devices out there isn't good for anyone.

It shouldn't be unreasonable to assume that a smartphone should be good for 5+ years.

[voltagex_]

Place the blame squarely at the feet of vendors like Qualcomm - they won't update the BSPs.

[kuschku]

The big issue with this is the – afterwards you always know better – completely stupid decision of Torvalds to never support a stable API for kernel modules, or drivers.

A similar issue happened recently between AMD and the kernel maintainers (AMD wanted a stable API for GPU drivers that would allow the same drivers to run on Windows, Mac, desktop Linux, and Android; and AMD had already built this API and was willing to maintain it).

In the end, this can only be solved if the Linux kernel gets a stable driver API, and with Oreo, Google added exactly that for their own fork of the kernel.

[majewsky]

> The big issue with this is the – afterwards you always know better – completely stupid decision of Torvalds to never support a stable API for kernel modules, or drivers.

It's not stupid. APIs need to change eventually. Even famously backwards-compatible Windows has its fair share of API changes. And when public APIs change, you can either drop support for the old one (thereby also dropping support for old hardware), or provide a compatibility mapping from the old to the new API (if that is even possible).

Linus knows that he does not have the manpower to do either. All he can realistically do is only support the current API, and require devs who change the API to update all usages inside the kernel source tree when doing so.

[kuschku]

There’s several problems with this.

(a) all drivers live in kernel space, even the sketchy drivers you have to download from NVIDIA

(b) if a driver wants to be included easily, it needs to become part of the kernel source tree, but that can only happen if Torvalds and his maintainers get full control.

(c) the API is broken with every release

As a result of all of this, the kernel maintainers refused even in any way to cooperate with AMD on their open driver (AMD needs a stable layer at some point to run the same additional functionality on windows, mac, and linux – the alternative is no linux support), we get ancient drivers on Android, with Google building their own HAL, and more shit.

At the same time, the syscalls, where maintaining a stable API is completely irrelevant and could simply be done via a small userspace library that you call to instead of doing actual syscalls, and which also massively would improve security if the translation between old and new syscalls would happen in userspace, is the one thing Torvalds maintains in the kernel.

The decisions made by Torvalds are reckless, massively hurt security and usability of open source, and the entire concept of open source ("we’d rather have only a proprietary AMD driver than an open AMD driver that relies on an HAL").