What’s the fundamental reason why SOTA modems are more challenging than microprocessors for Apple? Intuitively I would’ve guessed the opposite was true.
A long time ago i used to work at a company that made mobile phones. This was in the days before smartphones, and 3G was a shiny new toy. Back then, many companies would implement the protocols in the phone firmware, as opposed to today where chips usually offload a large part of this.
Despite doing our best to implement the GSM Standards[1], the thing at the time was 19,000 pages long, and one of our biggest issues was that our protocol stack would work well with most cell towers, there would always be some carriers that had configured their network just a tiny bit different, and we had to tweak our software to match that.
At the time, there were 5 major vendors of GSM cells, and none of them had intepreted the standards identically, so we would also have tweaks to allow for different vendors.
I don't remember exactly how many people we employed in our protocol division, but it was more than 100.
I do remember that implementing the first version of bluetooth took 19 people almost 2 years, and that, at the time, was a much simpler protocol (it probably still is). I can only imagine the "horrors" that have gone into 4G and 5G since then.
These days, all of the above problems are "solved" by simply installing a chip. Qualcomm handles any vendor hardware communication issues, and they're probably "big enough" to also have an impact in the opposite direction, making for a more streamlined protocol landscape.
This. The standard is also truly horrible to read. It's not really designed to be read, as much as it is an after-the-fact description of what somebody has already done.
To the point that testing against a very wide variety of base-stations (and the multitudinous variety of configuration options in those base-stations) is mandatory.
Which is presumably hard to impossible without flying around the world with test chips and physically being in those places, as I guess only a handful of carriers can and will reveal all their config settings and software stacks in practice.
The more I think about this modem development problem the harder it seems to get. No wonder Apple have struggled and nobody else even seems to try. The amount of tribal knowledge and random hotfixes in the Qualcomm firmware alone must be irreplaceable.
> Which is presumably hard to impossible without flying around the world with test chips and physically being in those places
We had people driving around with our phones logging to a laptop. They basically drove all over the country, trying to cover as much land as possible, and the phone(s) would then attempt to connect to different base stations.
The logs from this would be sent back to the developers to investigate failures, and somewhat often a bug would only manifest itself on a single cell tower.
Qualcomm has presuably done this work for most of the world, and has subsequently become the benchmark that telcoms calibrate against before deploying cell towers, leading to a more uniform protocol landscape.
If Apple hopes to create a new in-house modem chip, they will either need to calibrate it against Qualcomm, or do the gruntwork of travelling the world. In either case, i'm betting that just using Qualcomm chips regardless of the price will be better from an economic perspective.
Furthermore, everything GSM is covered by patents of the "big 5" (Motorola, Nokia, Siemens, Ericsson, I forgot the 5th), which at least at the time had free use of the others patents regarding GSM, but *everybody* else implementing GSM hardware and/or software must pay license fees. Qualcomm and Nokia had a big fight over this 3-4 years ago.
>>We had people driving around with our phones logging to a laptop. They basically drove all over the country, trying to cover as much land as possible, and the phone(s) would then attempt to connect to different base stations.
Mobile carriers still do this type whether via contractors or their own staff. Unfortunately, there's nothing that beat boots-on-the-ground field testing with actual devices (plural), which is already notoriously unreliable and prone to noise.
It's especially tough in countries with a large landmass (e.g. Canada, USA, Australia, Russia, China, Brazil, etc.).
One good thing that came out of it was the low power “GPS” on early smart phones, the ones that would triangulate the phones position from what cell towers it could “see”. That would not have been possible without someone driving around “everywhere” and recording GPS location alongside cell signal strength.
As far as i know it is less frequently used today than it was a decade ago, but i could be wrong. GPS in smartphones have moved from technology[1] to something you just use without thinking about how it works.
> We had people driving around with our phones logging to a laptop. They basically drove all over the country, trying to cover as much land as possible, and the phone(s) would then attempt to connect to different base stations.
Sounds like Apple needs to put test phones inside the cars doing “street view” as well as have Apple Store employees test them.
> and somewhat often a bug would only manifest itself on a single cell tower.
I can easily understand different brands/products/generations having their quirks, but I'm struggling to imagine what the source of uniqueness could be for a single cell tower.
Is it a huge variety of config options, that certain combinations of settings turn out to be rare? Or is it literally just something broken like malfunctioning hardware?
> but I’m struggling to imagine what the source of uniqueness could be for a single cell tower.
GSM is/was a complex beast. Each base station can only handle 8 simultaneous phone calls (the old 2G/3G multiplexed ones, not modern VoIP), so in crowded areas they’re usually configured with a very short range. Some large conferences have had base stations with their range measured in single digit meters (<30 ft).
Furthermore, like WiFi, bandwidth is limited, so base stations are deployed in a “beehive like pattern”, like a triangle with a base station radiating out from each leg, and broadcasting at different frequencies to it’s neighbors.
That alone leaves a lot of room for configuration errors on each individual base stations, but when i say “a single cell tower”, i meant on that drive. The bug might be with a specific firmware version of that base station manufacturer, or that particular hardware revision, or simply a configuration error, or maybe it was a bug in our software and/or radio firmware. There are a lot of “moving parts” that needs to be investigated, but from a developer perspective, the error only occurred on one base station.
It could of course also turn out to be a “broken” base station, and often enough we would fail to find the error, and had to contact the network operator to get them to help trace down the error.
Cellular base stations have lots of configurables. There's a lot of timing related settings in cellular radios. So you've got a thousand places to introduce a 500 Mile Email problem[0]. Some timing settings can run up against timing consts in the client firmware.
For instance towers around an airport might have some bands running with reduced power or disabled. This reduces cell size/overlap on those towers requiring more frequent handoffs between those towers. A client firmware might have a bug or unrealistic const set that fails in that frequent handoff situation. So it's a bug that only happens on some 5G bands near an airport but only during the summer because the more arid conditions increase microwave propagation by 1dB.
Huawei invented most of 5G technology and could tailor the 5G standards to their implementations. Which is also why they were first to market with carrier equipment [1] and had the first CPU with an on-chip 5G modem [2].
No they didn’t. This is utter garbage. I have worked in cellular testing since 1997. The first to market with pre-5G were NTT DoCoMo and Verizon. Huawuei were late to the game. My source is the companies we sold to and I supported personally.
If I ran a company where someone tried to release a 19K page spec I'd keep firing people until someone could produce a spec that I could actually review and sign off on, that could also still be reproduced in working technology. How does this kind of thing fly at all?
The thing is, GSM is not a single specification. It is layered like the OSI model, and each layer has multiple specifications for different subsystems.
When GSM was originally specified, there was serious doubt if it would work at all on the hardware available at the time.
A GSM call has a 3.62us send window, a 3x3.62us waiting period, and a 3.62us receive slot. That’s probably hard enough to achieve on a single cell, but factor in distance to the tower as well, and the handset has to do a lot of calculations.
Once you start considering handoff between cells things get even more interesting. The handset continuously reports a list of cell towers it can “see” along with the strength of their signals from the handsets position.
Once the cell tower decides that the handset is moving out of range, it propagates a handoff message to its upstream node, which repeats this process until a suitable downstream node is found, and each cell tower is then alerted that the handset with an ongoing call is switching from/to that cell, and finally the handset is informed. The handset has no say in this process other than the list of cell towers.
All of this took place in the late 1980s, it was finalized in 1987, and modern processors at the time would be the Intel 386DX running at 12 to 40 MHz. Obviously not handsets came with a 386 processor, and when I worked on mobile phones in the early 2000s, the norm would be something like an 8 to 12 MHz 16bit platform.
Since then the specification has been revised hundreds of times. In the original specification, messaging (SMS) was an afterthought, and a way of utilizing otherwise unused bandwidth, which was normally reserved for command and control. Turns out messaging was a hit with GenX, so many changes were made to that subsequently, like EMS[1], MMS[2] and RCS[3].
Likewise focus shifted from phone calls to data, which was 4G, and to IoT and always on devices, which is what 5G is about. 4G and 5G also has increased the number of active devices possible.
Add to that WiFi calling and all the other little enhancements, and I wouldn’t be the least bit surprised if the specification is closer to 50,000 pages today.
Thanks for taking time to write such detailed replies. I realize my reply was off the cuff; it's just astonishing that that much information was able to be absorbed and operationalized. Hats off to folks that were able to make it happen at all!
Analog is hard, and quite different from digital.* And then you have to work around the patent thicket in an already difficult domain.
* Nowadays clock speeds are so fast that you have a lot of analog things to worry about with all these high speed serial lines and on the chips themselves. But the domain constraints are in general simpler, with recovery and reconstruction opportunities available due to things like how you structure your protocols. The same is true in radio (came from radio, actually) but when you have a device operating at arbitrary and continually changing orientations, with all sorts of unknown environments outside you get all sorts of unpredictable multipath, loss, ringing and innumerable other problems to deal with.
> Modem chips are trickier to make than processing chips because they must work seamlessly with 5G wireless networks, as well as the 2G, 3G and 4G networks used in countries around the world, each with its own technological quirks. Apple microprocessors run software programs designed solely for its iPhones and laptops.
I would guess lack of expertise? Apple purchased PA Semi a while ago and the team went on to work on their processors. Have they purchased a company specialized in mixed-signal circuit design? That's one of the key differences, a modem contains a lot of analog circuitry and analog is very different from digital.
> Have they purchased a company specialized in mixed-signal circuit design?
They bought Intel's modem business, which was originally part of Infineon.
Intel had working 4G modems which were only a year or two behind Qualcomm to the point where most users wouldn't notice. But they never seemed to manage to make the jump to 5G.
It may be, but all new network capacity is being built on 5G so a phone without it will suffer more and more in performance, especially in crowded areas/events.
5G is absolutely insane. 100Mb/s for 20 USD/mo on a mobile phone in Hong Kong is absolutely mind-blowing. I suppose in low density areas like Tokyo, New York or Paris, it might be slower in some patches ?
What are you doing on a handheld device screen that would blow your mind once the novelty of seeing the big numbers on speedtest app wore off? Genuinely curious- what can you do on 5G phone that you can’t on your 4G phone?
A reasonably frequent usage of this speed is for families to switch over to mobile internet as the primary service for your home, having multiple people watch "TV" and videoconference all while someone is gaming (or dowloading a 40 Gb game) all through a single 5G modem.
I typically get around 300-600 Mbps on 5G, but in some areas I've seen speeds up to 1.3 Gbps (on NR, not even mmWave). This is in a Japanese city of pop 300,000
PA Semi was 2008 or so, and Intrinsity in 2009-10. In both cases they came with (more or less) working products, an existing relationship, and coherent engineering teams with an active development pipeline. Its fair to say the intel modem folks have a lot of that individually, but Im not clear how much of the whole transferred over.
Edit: and the HwEng org already had established VLSI, SoC, and PCB groups shipping products before those acquisitions happened.
IMO cellular network protocols are remarkably challenging to understand. Then you add to that various quirks of certain network operators, optimal power consumption and throughput, security.
They probably want their design to work as well as the baseline one, and globally.
From the article: "Modem chips are trickier to make than processing chips because they must work seamlessly with 5G wireless networks, as well as the 2G, 3G and 4G networks used in countries around the world, each with its own technological quirks. Apple microprocessors run software programs designed solely for its iPhones and laptops."
Also: "[Apple's modem] chips were essentially three years behind Qualcomm’s best modem chip. Using them threatened to make iPhone wireless speeds slower than its competitors."
I wonder what some of the regional quirks are, instances where the implementation strays from the standard, but if you don’t do it in quirks mode, you can’t connect to the cellular connection or calls will not go through etc.
A sort of woops, non-strict YAML moment, but in telecommunications.
The Apple CPU's architecture is already known (ARM). To iterate from there is probably easier than designing a modem without the basics already designed and ready to go.
Apple also has a pretty robust ability to dictate to their developers how and what they must do to run on MacOS, and they produce the compiler and toolchain as well. They seem to do a lot less shimming and other compatibility work on behalf of developers. Modem doesn't work that way, there's tons of existing, immutable hardware across the world with its own quirks and edge cases, all of which you have to support seamlessly.
>Intuitively I would’ve guessed the opposite was true.
It wouldn't be intuitive if people actually have some basic understanding of how modern wireless network works. Unfortunately even those who work on Hardware, CPU design and embedding programming dont understand that. Let alone 99.99999999% of HN.
Probably a mix of things. QCM has a lot of IP/Patents in this space so there are some "intuitive" engineering solutions that ... can't be pursued w/o a license.
But even if there was no legal blockers, RF engineering is a _whole other ball game_ compared to the "routine" circuits used in CPUs. I don't just mean from a "well, the schematic sure looks different" perspective, I mean from a "how you build it in the fab" perspective as well.
TL;DR: if electricity is magic, RF is black magic.
Despite doing our best to implement the GSM Standards[1], the thing at the time was 19,000 pages long, and one of our biggest issues was that our protocol stack would work well with most cell towers, there would always be some carriers that had configured their network just a tiny bit different, and we had to tweak our software to match that.
At the time, there were 5 major vendors of GSM cells, and none of them had intepreted the standards identically, so we would also have tweaks to allow for different vendors.
I don't remember exactly how many people we employed in our protocol division, but it was more than 100.
I do remember that implementing the first version of bluetooth took 19 people almost 2 years, and that, at the time, was a much simpler protocol (it probably still is). I can only imagine the "horrors" that have gone into 4G and 5G since then.
These days, all of the above problems are "solved" by simply installing a chip. Qualcomm handles any vendor hardware communication issues, and they're probably "big enough" to also have an impact in the opposite direction, making for a more streamlined protocol landscape.
[1]: https://www.etsi.org/standards/get-standards#page=1&search=&...