[danpalmer]

I remember reading in the UK government's security assessment of Huawei that one of the issues is not necessarily data being sent to bad places or backdoors in the software, it's that the engineering processes behind these devices/software are completely unable to protect against any sort of supply chain attacks.

The sorts of things they highlighted were: no version control, no code review, production builds happening on arbitrary machines, no automated testing, poor access control on code, no audit trail on code changes, the list goes on, and that's just for the software side. The conclusion was that Huawei were about a decade away from being able to even claim they had no backdoors. And that's a major telecoms hardware provider, trying to sell into governments and major infrastructure projects.

I'm not in the least bit surprised that TP-Link are doing this, and also not at all surprised that when questioned on it they are (so far) unable to actually describe why it's happening or really seem to know anything about it.

I think this sort of product is built in a very different environment to what most HN users would expect.

[user3939382]

I had/have a Gemini (Android) from Planet Computers. I disabled wifi and forced its network connections through an ethernet adapter that I connected to a mirror port->wireshark and through a proxy after putting in my own root certificates.

My goal was to silence its network activity when I wasn't using it. One by one I removed APKs and blackholed IPs and domains, starting with everything from Google. I was disturbed to discover that, even having nothing installed and everything ripped out that I could, once every week or two while sitting untouched it would phone home to an IP address in China that I failed to connect to any software on the phone and whose IP WHOIS made no sense. I asked Planet Computers about it and they had no idea.

[landemva]

Would like to see this writeup somewhere, both for the results and for the methodology.

[servytor]

The UK has been indecisive about trusting Huawei [0 (sorry for the Daily Mail link),1,2,3]; I don't know why and find it very interesting. I am used to reading about nation states having an unwavering opinion, not flipping back and forth (unless because of political lines). They claim that their hand has been forced by the USA [4].

[0] https://www.dailymail.co.uk/news/article-7935905/MI5-MI6-GCH...

[1] https://www.cnbc.com/2019/10/09/former-uk-spymaster-john-saw...

[2] https://www.reuters.com/article/us-britain-huawei-tech-five-...

[3] https://www.ft.com/content/90c07bbe-38ce-11e9-b856-5404d3811...

[4] https://www.euractiv.com/section/politics/short_news/uk-bann...

[blibble]

vince cable is not part of the government and has no more information on the situation than you or I

the government was resistant to the US' position until the CCP's crackdown on hong kong, at which point they reversed their position

[danpalmer]

Yeah the actual position of the government is all over the place and it's all tied up in the politics of US/China relations.

I think the findings in the report are still a concrete assessment of Huawei's abilities that we can draw conclusions from about their product security.

[geokon]

A perfect situation for Hanlon's razor...

Is this stuff not par for the course? Everything hardware/embedded in my experience is like a decade or two behind the current norms for c/c++ programming. What I never understood from that audit, was this code quality unusual? I didn't get the sense they audit European and American companies - so sure they looked at the source and said "lol your code sucks" but there was no baseline for comparison

But it sounds like you know the situation better - maybe you have better context. I've been curious to know from someone more familiar with the subject

[danpalmer]

> A perfect situation for Hanlon's razor...

I'm not at all suggesting that Huawei (or TP-Link, or anyone else) are actively attempting to subvert security systems or intentionally adding backdoors. In that sense it's probably right to conclude this is ignorance.

The problem is that an attacker, especially those with the backing of a nation state, can trivially attack those insecure supply chains and install backdoors or data exfiltration.

As for whether others are as bad, I think the sort of audit that was done on Huawei is done for other companies attempting to sell into that level. These audits are not really about looking at the code – sometimes they do, but you're never going to get a useful security audit of 10s-100s of millions of lines of code. They're more about the security posture of these companies, and in that way, Huawei failed.

I do expect that Cisco, HP, other network hardware vendors are better at this. Do they still have crap code? Sure. Do they still have security vulnerabilities? Of course. Could a nation state still get a backdoor in? Probably. But would it be significantly harder to do, easier to detect, and easier to resolve? Yes, and that makes them better suited to critical infrastructure.

[geokon]

"But would it be significantly harder to do, easier to detect, and easier to resolve? Yes, and that makes them better suited to critical infrastructure. "

But like what is that conclusion based on?

I'm not saying you're wrong - just curious why you hold HP and Cisco in high esteem.

At least in terms of engineering talent I'd expect them to be much worse. Huawei is prolly the Google of China paying huge salaries and getting the county's top engineers (along with Alibaba). When I lived in Santa Barbara Cisco didn't have a good rep and they didn't pay well. A typical bureaucratic officespaceesque soul sucker. I don't know about HP but I don't get the sense it's a presitgious place to work either.

Again, these are very shaky ill informed judgments on my part I admit :) hence why I'm curious if you're talking from a position of knowledge on the subject

[danpalmer]

> But like what is that conclusion based on?

It's based on a few assumptions, but ones I feel are reasonable to make. The fact these companies will have been audited in the same way, but that the concerns have not been raised (by government, industry, security consultants) suggests that these processes are very different.

Version control, code auditing, code review, reproducible builds, etc, those will all contribute to being able to protect against attackers.

You're right that there's a huge talent pool in China, and there is good engineering happening in China, but there are also cultural barriers to it in some places. The 9/9/6 working culture in Chinese tech companies optimises for throughput not quality, and the general impression I have from reading about internal engineering cultures at other Chinese tech companies aligns with the Huawei report.

I'm not speaking from a position of expertise, I am judging this and drawing my own conclusions, but I don't feel they are ill informed (nor do I think yours are). I'm confident in the facts I know, have evidence for my opinions, and have reason to believe my suspicions.

[geokon]

"The fact these companies will have been audited in the same way .."

Have they? Are you sure? The Huawei audit were not a routine audit. According to Wired it was done by the special British "Huawei Cyber Security Evaluation Centre". I can't find any evidence the UK National Cyber Security Centre has done the same with Cisco or HP.

> am judging this and drawing my own conclusions, but I don't feel they are ill informed (nor do I think yours are)

The difference between us is that I definitely think MY conclusions are ill informed. Hope someone who knows what they're talking about can chime in

[Dah00n]

It all sounds very reasonable untill you remember that multiple backdoors and hardcoded hidden admin accounts have been found in Cisco products. I have yet to see any proof that Huawei are worse (or better) than Cisco. IMO absolutely nothing have been proven in terms of quality versus other manufacturers outside of political standpoints in all this. As far as I can tell this audit have not been done (or at least not published) to any other manufacturer than Huawei. It's 100 % politics and zero evidence of quality when only one side gets tested and published.

[legalcorrection]

Also a perfect environment to slip in back doors that look like mistakes.

[danpalmer]

Exactly, that was their point. I suspect there are a lot of software supply chains that are actually compromised, because it's just too easy when this is the standard of software engineering.

It's easy to forget that git for example is not just a big "undo" button, it's a cryptographically secure audit log of all changes made, that allows you to know exactly what software you're actually shipping.

[bostik]

> cryptographically secure audit log of all changes made, that allows you to know exactly what software you're actually shipping.

To be pedantic: not quite.

Git is certainly a cryptographically secure audit log[ß] but it only tells you what the source code was that went _into_ the build at the time of checkout. You can subvert the process through malice (eg. Solar Winds), through incompetence (eg. off-tree "magic" patching as a build step), or through sheer negligence.

Reproducible and auditable builds are a much harder problem than source code provenance.

ß: from my previous job: once auditors understood what git is, they loved it. They, by their profession, love immutability. Failing that, they consider tamper-evidence a really good second-best.

[danpalmer]

Of course, I was simplifying this somewhat to make a point. I think git can be a significant part of the solution, but there's a lot more that goes into it.

[bell-cot]

With enough mistakes, why would anyone bother to install a back door? That costs money, and reduces deniability.

[Dah00n]

Yes, it has happened multiple times at Cisco.

[_wldu]

I find it hard to believe that Huawei does not use version control. No company is perfect, but surely software developers (at a multinational company) with advanced degrees in CS and ECE are using version control.