[mike_hearn]

Which is presumably hard to impossible without flying around the world with test chips and physically being in those places, as I guess only a handful of carriers can and will reveal all their config settings and software stacks in practice.

The more I think about this modem development problem the harder it seems to get. No wonder Apple have struggled and nobody else even seems to try. The amount of tribal knowledge and random hotfixes in the Qualcomm firmware alone must be irreplaceable.

[8fingerlouie]

> Which is presumably hard to impossible without flying around the world with test chips and physically being in those places

We had people driving around with our phones logging to a laptop. They basically drove all over the country, trying to cover as much land as possible, and the phone(s) would then attempt to connect to different base stations.

The logs from this would be sent back to the developers to investigate failures, and somewhat often a bug would only manifest itself on a single cell tower.

Qualcomm has presuably done this work for most of the world, and has subsequently become the benchmark that telcoms calibrate against before deploying cell towers, leading to a more uniform protocol landscape.

If Apple hopes to create a new in-house modem chip, they will either need to calibrate it against Qualcomm, or do the gruntwork of travelling the world. In either case, i'm betting that just using Qualcomm chips regardless of the price will be better from an economic perspective.

Furthermore, everything GSM is covered by patents of the "big 5" (Motorola, Nokia, Siemens, Ericsson, I forgot the 5th), which at least at the time had free use of the others patents regarding GSM, but *everybody* else implementing GSM hardware and/or software must pay license fees. Qualcomm and Nokia had a big fight over this 3-4 years ago.

[otoburb]

>>We had people driving around with our phones logging to a laptop. They basically drove all over the country, trying to cover as much land as possible, and the phone(s) would then attempt to connect to different base stations.

Mobile carriers still do this type whether via contractors or their own staff. Unfortunately, there's nothing that beat boots-on-the-ground field testing with actual devices (plural), which is already notoriously unreliable and prone to noise.

It's especially tough in countries with a large landmass (e.g. Canada, USA, Australia, Russia, China, Brazil, etc.).

[8fingerlouie]

I had assumed it was a lot less these days.

One good thing that came out of it was the low power “GPS” on early smart phones, the ones that would triangulate the phones position from what cell towers it could “see”. That would not have been possible without someone driving around “everywhere” and recording GPS location alongside cell signal strength.

As far as i know it is less frequently used today than it was a decade ago, but i could be wrong. GPS in smartphones have moved from technology[1] to something you just use without thinking about how it works.

[1] https://www.azquotes.com/quote/343497

[lstamour]

> We had people driving around with our phones logging to a laptop. They basically drove all over the country, trying to cover as much land as possible, and the phone(s) would then attempt to connect to different base stations.

Sounds like Apple needs to put test phones inside the cars doing “street view” as well as have Apple Store employees test them.

[crazygringo]

> and somewhat often a bug would only manifest itself on a single cell tower.

I can easily understand different brands/products/generations having their quirks, but I'm struggling to imagine what the source of uniqueness could be for a single cell tower.

Is it a huge variety of config options, that certain combinations of settings turn out to be rare? Or is it literally just something broken like malfunctioning hardware?

[8fingerlouie]

> but I’m struggling to imagine what the source of uniqueness could be for a single cell tower.

GSM is/was a complex beast. Each base station can only handle 8 simultaneous phone calls (the old 2G/3G multiplexed ones, not modern VoIP), so in crowded areas they’re usually configured with a very short range. Some large conferences have had base stations with their range measured in single digit meters (<30 ft).

Furthermore, like WiFi, bandwidth is limited, so base stations are deployed in a “beehive like pattern”, like a triangle with a base station radiating out from each leg, and broadcasting at different frequencies to it’s neighbors.

That alone leaves a lot of room for configuration errors on each individual base stations, but when i say “a single cell tower”, i meant on that drive. The bug might be with a specific firmware version of that base station manufacturer, or that particular hardware revision, or simply a configuration error, or maybe it was a bug in our software and/or radio firmware. There are a lot of “moving parts” that needs to be investigated, but from a developer perspective, the error only occurred on one base station.

It could of course also turn out to be a “broken” base station, and often enough we would fail to find the error, and had to contact the network operator to get them to help trace down the error.

[k0k0]

> Each base station can only handle 8 simultaneous phone calls

Perhaps you are confusing that each GSM transceiver (TRX) provides 8 time-division channels with call capacity, but most cells and specifically the BTS in GSM parlance, especially any in a well populated area have/had way more than one transceiver. 30-40 was not unheard of in later equipment, though 10 or so was more typical. Late in GSM's life there was another technique to squeeze more channels, OSC.

Furthermore those 8 TDMA slots could be split in 1/2 or 1/4 with lower rate codecs, so it was more than 8 per TRX as well.

anything bigger than a picocell would carry way more than 8 simultaneous calls.

> Some large conferences have had base stations with their range measured in single digit meters (<30 ft)

Femtocells are still a thing today. Not so much to do with frequency capacity.

[8fingerlouie]

> Perhaps you are confusing that each GSM transceiver (TRX) provides 8 time-division channels with call capacity

I was talking 2G/3G. It has been 20 years since i wrote software for mobile phones :) But i should probably have been more specific and said you can have 8 active calls per channel. And i have no doubt today with 4G/5G and a shift to VoIP that you can have way more than that.

> Femtocells are still a thing today. Not so much to do with frequency capacity.

I was attempting to refer to the scalability/complexity of the network, which scales from 10s of kilometers to 10s of meters, or even more/less.

But as i said, i wrote software for the phones. My understanding of the network side of things was/is limited to the rudimentary knowledge needed to write that, and i'm happy to be corrected.

[giantrobot]

Cellular base stations have lots of configurables. There's a lot of timing related settings in cellular radios. So you've got a thousand places to introduce a 500 Mile Email problem[0]. Some timing settings can run up against timing consts in the client firmware.

For instance towers around an airport might have some bands running with reduced power or disabled. This reduces cell size/overlap on those towers requiring more frequent handoffs between those towers. A client firmware might have a bug or unrealistic const set that fails in that frequent handoff situation. So it's a bug that only happens on some 5G bands near an airport but only during the summer because the more arid conditions increase microwave propagation by 1dB.

[0] https://web.mit.edu/jemorris/humor/500-miles

[Goodroo]

Well Huawei seems to have succeeded

[reic]

Huawei invented most of 5G technology and could tailor the 5G standards to their implementations. Which is also why they were first to market with carrier equipment [1] and had the first CPU with an on-chip 5G modem [2].

[1] https://www.wired.com/story/huawei-5g-polar-codes-data-break...

[2] https://en.wikichip.org/wiki/Kirin_990

[justcradd]

No they didn’t. This is utter garbage. I have worked in cellular testing since 1997. The first to market with pre-5G were NTT DoCoMo and Verizon. Huawuei were late to the game. My source is the companies we sold to and I supported personally.