I am a big fan, but I also find this situation and the response to be a bit disappointing.
I highly respect Martijn and the effort he has poured into pmOS and Pine64 and surrounding projects. Without wanting to make other important people feel left out, I consider him an absolute legend and the fact that he has felt compelled to write this blog post (and, unfortunately, to leave) is massively concerning.
I think it's very possible that there has been no malice here and it's all just an accident of one thing leading to another, but something has gone awry here and I think it would be nice to hear what Pine intend to do to put it right, or to know that there is such a plan at all.
> As for the reason why Pinebook Pro doesn’t ship with a bootloader on SPI – it isn’t caused by favoritism for a particular group of developers or disregard for good ideas, but rather something more trivial. Namerly, a functional Tow-Boot build for the Pinebook Pro wasn’t available at the time of manufacture and shipping.
One particular point I want to make: I'm willing to believe Pine about the above, but it's too late to hear about this after crucial people have been miffed. This seems to indicate that communication with the various projects has not been good. This seems like a crucial thing to improve on to prevent this recurring.
I don't have enough knowledge of the situation and am incapable of making any judgements as to what is really going on, but I sincerely hope that Pine will consider this situation carefully and whether something can be done to put things right again.
I really wish to see Pine and their hardware do well and flourish. That was never going to be easy, but it will be even less easy with the loss of people like Martijn.
There has been zero communication towards Tow-Boot developers during this process, nothing about the issues on the new batch. The moment Tow-Boot learned about this is when people received the new Pinebook Pro and it came with a different bootloader.
I found logs. For context, tl_lim is the founder of Pine64.
2022-06-01, the Pine64 dev IRC chat:
> [T] <tl_lim> I pause on move forward due to no one commit to maintain. If MayueIC interest to maintain, lets move forward and include Tow-Boot into the PBP factory build. @MayeulIC, you can send an email to info@pine64.org, we will ship a free Pinebook Pro to you so that you can maintain the Tow-Boot.
> [T] <tl_lim> On the integration, just follow the recent PinePhone Pro way that flash OS build to eMMC then flash Tow-Boot to SPI Flash.
> [T] <tl_lim> Frankly speaking, I am happy on developer commit maintaining Tow-Boot on Pinebook Pro. Thanks and a big shout out to @MayeulIC.
2022-06-02, in the Tow-Boot chat, which was discussing tl_lim's above comment:
> spikerguy: Tllim asked on. 25th April about pbp tow boot support i replied it just work fine as it should. But just because i am the one to work on factory images plus i am the manjaro arm pkg maintainer maybe he didn't wanted to hear from me instead the community that was assuring him about tow boot and none replied
> The current batch factory image is already submitted. So hopefully we can keep things ready for the next batch
I don't see all comms, but overall I see Tow-Boot on SPI at factory-time as nice-to-have but not essential (users can easily do this, distros don't need to require it, and the next batch can improve it if needded), Pine64 has showed willingness to use Tow-Boot. Tow-Boot has 1 developer, and I'm not sure if they were willing to commit to support this. I understand if they weren't.
This issue is blown out of proportion.
Maybe a good step would be for Pine64 to fund Tow-Boot's support of Pinebook Pro.
This is just after I left. And after me saying (in not the public channel) that samuel is maintaining Tow-Boot, other developers are helping (like me) but that didn't matter.
This is a disappointing response. I don't have a horse in this race (besides wanting to have access to good hardware for FOSS platforms), but I do have my ear on the ground in this community, so here are my thoughts on this.
> [removing SPI] was based on the fact that for years SPI was largely unused on PINE64 devices.
People have been arguing that a u-boot firmware needs to be installed on that flash chip for years, too. Essentially everyone in the dev community has been pushing them to do this for essentially the entire lifetime of their product line.
Would you rather find, download, and flash a special-purpose, pinebook-specific image onto a microSD card, pop it in, and boot that up to install a distribution, or would you rather it supports a standard UEFI booting interface over USB, netboot, whatever else, like every other laptop can?
> We created a space for development talks to be held (as we always had)
My understanding from speaking to friends and colleagues who work with Pine64 is that these spaces are pretty toxic. Manjaro has the loudest voice and other projects don't get the respect they deserve. And I'm prepared to believe it if this flippant response quoted in Martijn's article is accurate: "people who want [an SPI chip] can just solder one on."
This is not the behavior that I expect from a project which claims to honor its community.
I'm very disappointed in this response and in PINE64 generally. I expected an apology and a promise to improve, and this isn't it. Pine, I believed in you from the start.[0] I thought you were better than this. The past few years of your behavior has disillusioned me to your project and I can no longer offer you my support.
So as is practically stereotypical for the Linux community, we're going to let a squabble over distros send us to buying proprietary hardware and more than likely... straight up giving Google more money, as seems to be the top hardware choice of vendors of so-called "secure Android forks".
Me too, I was hoping to get a Linux phone at a price that match the current lack of polish (for me the pinephone had that price) but Linux means community and diversity for other OS'es to me. So if they don't work with the community, then half the value of the phone disappears...
I will now consider buying second hand phone with postmarketOS support. It means I'll have to accept proprietary hardware but, as it will be second hand, it means I'll be more in line with my eco-environemental-whatever-half-baked principles.
Nobody cares, I just wanted to illustrate what are the reasons I would like to buy a Pine64 and how those reasons revolves around some basic principles and trade off's.
You could also get a second-hand PinePhone (just buy postmarketOS CE or later to avoid bad hardware bugs).
Regarding other postmarketOS supported devices, I quite like the Xiaomi Pocophone F1 and the OnePlus 6 - but be aware that bootloader unlock on the Xiaomi is a bit painful and charging on the OnePlus 6 is really slow (although that might have been fixed lately/be fixed soon).
It's an excellent soldering iron, hardware wise. Lightweight, heats up extremely fast, holds up heat very well. Never tried a custom firmware, but had no need either.
> Would you rather find, download, and flash a special-purpose, pinebook-specific image onto a microSD card, pop it in, and boot that up to install a distribution, or would you rather it supports a standard UEFI booting interface over USB, netboot, whatever else, like every other laptop can?
It's possible to write Tow-Boot to the eMMC to do exactly this. SPI isn't needed, it's just more convenient. I happen to prefer SPI and Pine64 was willing to go with SPI too.
I think this issue is blown out of proportion.
Pine64 listened to feedback. The problem here was that timeline's didn't match, not bad intention.
If you want to buy a phone that Just Works, buy a Samsung Galaxy or a Google Pixel.
You can also buy a Pine64 product and if you do not like the hardware you are free to modify it to your heart's content. However, they don't owe it to you to perform the modifications you want, certainly not if there is a shortage of the part you want them to install.
As an outsider with no insights (but owning a PineBookPro and considering a PinePhonePro), it surely looks like Manjaro is the king of the mountain - and that Martijn's blogpost is likely close to The Truth.
Listening and doing your own things anyways is the time-honored way of ignoring people...
Even as an Arch user (that hence "gets" Manjaro) that is sad :(
Much of the device still doesn't even work. And we're talking simple stuff like the cameras. Calling is also super sketchy. You can text but voice calls effectively is a fail.
The IMU does appear to work correctly, for things like rotation. It is pretty fast.
You can also fry your device (hardware damage!) if you get the pinephone keyboard and plug in USBC into the phone port instead of the keyboard port. You read that correctly. Hope you don't mess up if you're tired.
And if your battery on the PPP goes too low, the utter shit firmware goes into a boot loop and can't charge. You have to take off the case and hit the recessed button to put it in DFU mode. That bypasses the firmware causing bootloops. And it's slllloooowwwww charging.
I completely and utterly regret such a shit device. And I say that about the very hardware, AND the reticence in actually putting out base builds of working software.
The cameras are far from "the simple stuff", it's one of the more complicated systems in mobile devices with multiple high speed components from different vendors that have to work together and not very much documentation for the hardware existing.
Admittedly, that's doubly concerning. Pine chooses the BoM; we don't.
And if they're choosing unsupported/unsupportable chips without doing the barest of legwork to get them functional (or hell, get docs public), then that's a huge problem.
Pine should be showing on their pages a hardware matrix showing what works and what doesn't. The Pinephone Pro doesn't qualify for the definition "phone".
Martijn, I want to really give you some credit for these responses: It's rare to see someone who will step in and defend bad takes against the party they are also taking issue with.
> Pine should be showing on their pages a hardware matrix showing what works and what doesn't. The Pinephone Pro doesn't qualify for the definition "phone".
I'm actually on board, but do note that that's going to be a chart full of subnotes and caveats - things like "camera produces an slightly red-shifted but passable image on default software shipped from factory, red-shift is fixed with kernel 5.19-patch-foo but that breaks focus, Mobian ships that kernel with a userspace correction filter that fixes it in the latest nightlies[...]". You'll end up with a small table that's 90% little yellow "works with caveats" boxes and then several pages describing the caveats and various working and shades-of-working configurations. Oh, and it'll need to be timestamped and constantly updated (as in "major bugs fixed, and sometimes added, on a weekly basis")
In short, summarizing active development is never going to be simple.
If it’s no, but has a dozen asterisks, to the end user, that’s still a “no, this component does not work properly.” The footnotes can be 5 pages long but the chart is still straightforward
One day I’ll probably end up with a Linux phone, but I mean, not if I’m worried I’ll get surprised when it arrives
Rear sensor was chosen as a balance of kernel code available and picture quality, I had some influence there, front sensor was originally the ov5640 but that just got replaced due to availability.
Nobody wrote the driver for H.264 HW encoder support, yet. When that happens, Pinephone will be capable of that, too.
That's not a $200 proposition, though. :)
And I guess there's no huge motivation, given the expected quality of the resulting image. Maybe someone will write it for some other similar AW SoC that uses the same HW block.
If it's beta software and prototype hardware then they should be in the 'give it to some people willing to help with bugs and documentation' phase, not 'charge people for a phone that can't make phone calls' phase.
I used to be all for early adoption, but too many people ran off with my money via Kickstarter and Steam Early Access.
Your $5 bargain bin webcam doesn't work without the computer and the software that runs on it though. The PinePhone isn't the problem, the problem is the lack of software. It is ridiculous to compare the two things.
I’d expect my PinePhone to come with software too. I mean, I guess it did, I think I just expected more from the open source community than from the nameless fly-by-night vendor.
They claim that it supports eMMC booting natively. I bought 4 of each. And I bought a eMMC USB dongle to pre-liad the images.
And .... No booting off of eMMC. And utterly NO help from anyone. No guides, no nothing.
And that's hardware stuff that Pine attested would work. And it completely doesn't. I still have to boot using mSD at a whole class 10.
Their whole "plan" and company is a shit show. It's all gongkai crap. Using or buying will give you pain and suffering if anything goes even slightly wrong.
> The example given in the blog post supposes that the community team and Pine Store employees were firmly intent on removing SPI on the PinePhone Pro and coerced not to ship Tow-Boot.
I don't see where in Martijn's blog post that "coercion" was "supposed" at all. What he actually wrote:
> Negotiating this solution was hell. Manjaro is incentivized not to agree to this, since it cedes their sole control over the bootloader, and PINE64 listens to Manjaro before anyone else. Furthermore, PINE64 does not actually want to add SPI flash chips to their hardware. Apparently, there has been some issues with people using SPI flash as RW storage on the A64-LTS boards, which would be a support issue.
> After months of discussions between the community, Manjaro, and PINE64 leadership, we finally were able to convince them to ship the PinePhone Pro with an SPI flash chip with Tow-Boot installed on it.
He doesn't even claim that Manjaro objected at all to the idea of adding an SPI chip and preloading Tow-Boot on it; only that they have a possible vested interest in resisting it, and that PINE64 therefore also does for as long as it considers Manjaro the preferred OS for PinePhones.
I think the problem is that as an external observer, it's almost impossible to distinguish "we listen but don't/can't always act on it" and "we don't listen", because they produce the same actions and are only possible to tell apart by closely following the exact communications (which isn't always possible since some happen in private, and is unreliable anyways if some is trying to appear engaged while not actually listening).
> I may add that the decision was made among ourselves, without the input of any third parties or partners
Maybe this, right here, is actually a major part of the bigger problem and not exonerating like it seems to be intended
> The reasoning behind not including SPI on the PCB wasn’t motivated by any single project or outside source – it was based on the fact that for years SPI was largely unused on PINE64 devices.
So if they only talked amongst themselves, how could they have any idea of if the SPI was already being used or not?
> And yet, despite this, we agreed to include it on the PinePhone Pro because developers from multiple projects – postmarketOS being one of them – were adamant that it was an absolute necessity.
So yeah, sounds like if they had engaged earlier, a lot of unnecessary frustration and grief could have been completely avoided.
Unfortunately this post does not even hint at the more significant point in Martijns post, the one about the relationships between PINE64 and the community distributions.
I think I'm on Team Pine here. I don't really understand the problem Martijn has. It seems to boil down to that they agreed to everything he wanted, just not fast enough?
Sure, but that's not a deal breaker when you can just change it? I too don't particularly like Manjaro but here's a situation where he's doing damage to basically the only brand that even remotely supports his interests (and that would even listen to him personally) at a low cost. Or maybe there's other better phone options to run Phosh that I don't know about?
As a veteran embedded engineer, I was shock to see that another variant of boot loader is being recommended.
I’m quite sure that the bootloop is more than fixable but have my doubts that SPI is not that option.
Tow-wut? Sounds something like “extoll-embrace-engulf” and Balkanization to me. Not to stand in the way, but if that fixes the low-charge/bootloop problem then so be it but it doesn’t.
Still have the PinePhone Classic, still works, on my anti-static pad, but it may not serve us much longer.
Tow-Boot is actually a really exciting project that wants to make it so that distributions don't have to manage device-specific u-boot packages for ARM devices, and can instead provide generic images like they do on x86.
This right here. As much as I would love to see more faster and powerful ARM processors come to laptops and desktops in the future, I dread the current ARM bootloader situation coming along with it.
This seems easy enough, right? If the SPI chip is in shipped units, and tow-boot is available for it now, then it should be simple enough to ship out of the box starting with the next batch of hardware, and it should be simple enough to post official instructions for installing it from Manjaro. While that wouldn't change the past, it would fix things going forward and let Pine back their words with actions.
I get the feeling that Martijn's situation was very much death by a thousand cuts and that didn't quite get captured well by the focus on the SPI issue - which Pine64 seems to have defended reasonably well here.
I'm a bit confused by the whole bootloader situation. Locking down a bootloader on an open device seems quite bad, but what's the issue here exactly? The user can install any bootloader they want, can't they? I know the ARM bootloader system isn't anything like the PC bootloader system, but even if the bootloader is loaded from eMMC, in what way is the user impeded?
I would guess all the user should need to do to use their own bootloader is to pacman -R manjaro-arm-uboot and pacman -S pmos-uboot to hand over control of the bootloader image if you buy a device flashed with Manjaro, or overwrite the eMMC bootloader with one that manually tries to boot from the SD card before resuming normal boot?
It seems that the frustrations about Manjaro go further than just the bootloader but I don't really understand why a SPI flash chip would even be a necessity in the first place.
> even if the bootloader is loaded from eMMC, in what way is the user impeded?
The "bootloader" isn't just a bootloader, it's effectively a critical firmware component. If the "bootloader" is missing or garbled, the board is bricked and can only be recovered by removing the eMMC, which would be very difficult for most users. Such a thing should not be stored on a data partition and managed like any other distribution package. If they aren't going to ship a separate SPI chip, the eMMC should at least be repartitioned to include a separate "firmware" partition and the bootloader should be shipped there. Distributions should then boot via a chainload mechanism, just like on x86 hardware. (This mechanism can be UEFI based but need not be.)
As a veteran embedded engineer, I was shock to see that another variant of boot loader is being recommended.
I’m quite sure that the bootloop is more than fixable but have my doubts that SPI is that option.
Tow-wut? Sounds something like “extoll-embrace-engulf” and Balkanization to me. Not to stand in the way, but if that fixes the low-charge/bootloop problem but it doesn’t.
Still have the PinePhone Classic, still works, on my anti-static pad, but it may not serve us much longer.
I never said they didn't include SPI, it took an incredibly amount of arguing over several weeks to get it there on the PPP, then they again didn't want to ship it on the PBP. In the end the PBP does have SPI but still didn't ship with something flashed to it.
While they did in the end listen and add SPI _after_ manjaro finally agreed. that doesn't mean they listened to the whole community. Ofcourse PINE64 and Manjaro don't have to deal with how the other distributions boot.
Well the issue with this happening in private telegram channels is that this will now turn into a "he said" "they said" situation, none of this happened publicly and neither side can provide sources. At the time the argument was definitely not that the chips weren't available.
Instead of delaying the PPP production by one day to get U-Boot sorted they decided to not ship mainline U-Boot at all. Which was only possible because Manjaro provided them with an image to ship with that.
In the end the only response there is is a blog post that never claims anything is wrong or should happen differently, after acknowledging all that I've done.
I only wrote why I left, that they themselves think they're manjaro focussed because the hardware that can't ship Linux doesn't ship manjaro doesn't change that this whole behind the scenes situation really burned me out, and I left.
You have every right to leave, and obviously if you're feeling burnt out, you made the right call for yourself. But it is frustrating that it seems like a distro preference issue has expanded into what will probably end up selling more proprietary phones at the end of the day.
They ship devices with a completely broken OS[1], I could never recommend their products to anyone, except maybe someone who is a technical type and comfortable going through the process of installing another distro + happy to spend time doing that.
[1] At least as of a couple of months ago. I'm talking basic functionality being broken, like for example trying to add an icon to the home screen and the phone crashes. I encountered multiple bugs in just the first few minutes of usage.
To be fair, I had a very similar experience on my very first minutes with an iPhone on one of the .0 iOS releases. So many glitches and broken things, even on the onboarding process, many of them reliably reproducible. That said, when I grabbed that iPhone the next time and updated it a few minor releases further, most of them were fixed.
the difference is that there were perfectly good distros for the PinePhone, but then it was decided to start shipping it with a half-baked one. It's as if Apple went back from those later iOS releases to the earlier one.
The original "Why I left Pine64" article adds more info about what went on.
If they proved anything it was that they can’t build a phone. The world got a useless brick because politics. Whenever I read about the pinephone I somehow narrate it in the voice of Jordan Peterson
I highly respect Martijn and the effort he has poured into pmOS and Pine64 and surrounding projects. Without wanting to make other important people feel left out, I consider him an absolute legend and the fact that he has felt compelled to write this blog post (and, unfortunately, to leave) is massively concerning.
I think it's very possible that there has been no malice here and it's all just an accident of one thing leading to another, but something has gone awry here and I think it would be nice to hear what Pine intend to do to put it right, or to know that there is such a plan at all.
> As for the reason why Pinebook Pro doesn’t ship with a bootloader on SPI – it isn’t caused by favoritism for a particular group of developers or disregard for good ideas, but rather something more trivial. Namerly, a functional Tow-Boot build for the Pinebook Pro wasn’t available at the time of manufacture and shipping.
One particular point I want to make: I'm willing to believe Pine about the above, but it's too late to hear about this after crucial people have been miffed. This seems to indicate that communication with the various projects has not been good. This seems like a crucial thing to improve on to prevent this recurring.
I don't have enough knowledge of the situation and am incapable of making any judgements as to what is really going on, but I sincerely hope that Pine will consider this situation carefully and whether something can be done to put things right again.
I really wish to see Pine and their hardware do well and flourish. That was never going to be easy, but it will be even less easy with the loss of people like Martijn.