[_pplp]

Hoo boy, this is gonna be a fun one. For reference, I spent a year (mid-2018 to mid-2019) running the UniFi Network team and worked with Nick during that time.

> * Why was it so easy for a lead engineer to get access to a root AWS user without anyone else being notified? I.e. AWS GuardDuty provides FREE alerting for when an AWS root IAM account is logged in or used, this account should be under lock and key and when used, confirmed and audited by relevant persons or teams.

The "Cloud Lead" that Nick took over from gave zero fucks. He ran all the AWS stuff for Ubiquiti under his personal AWS account. Nick came in and started putting "proper" AWS structure and security in place, primarily by scaring Robert (the CEO) into giving him the keys to the castle (my own personal opinion of Robert is... not the greatest).

One thing to understand about Ubiquiti (at least during those times) is that the company had zero C-level execs. There was Robert.... and then nobody knows. I asked repeatedly why we didn't have a CTO, or a COO, or a CFO, or CMO or ANYTHING and I got nothing but shrugs and "idunno" as a response for the whole year I was there.

So when Nick came in, a very... let's just say "forceful" personality, he immediately won over Robert and ended up with carte blanche over pretty much all of Ubiquiti's cloud accounts. Which were basically... everything. All the UniFi Network services, UniFi Protect services, you name it. If it was connected to the cloud in any way, Nick had access to it.

So why wasn't anybody else notified? Simple. Because he was basically "god". If anybody was gonna be notified, it would've been Nick. He was the top of the totem pole company-wide when it came to AWS.

Also, for some perspective, at that time Ubiquiti kept all the hardware signing keys in a private GitHub repo that every employee had read access to. And they were in plain-text. So... yeah.

> * Furthermore on the root account being easily accessed, the root account in the companies I've worked at had MFA enabled, and the QR code is locked in a safe only accessible by two people agreeing it needs to be accessed in a break glass situation, where warranted.

See above for the quality of security processes and practices this company had in place.

> * Why was he also able to delete critical CloudTrail logs and reduce their retention to 1 day? I.e. These logs should be in a S3 bucket or other environment where such changes cannot be made. Alternatively, they should be shipped to a redundant service that manages this risk to prevent data deletion

See above. (re: "god") Nick answered only to Robert. And he'd already successfully hoodwinked him. He could do whatever he wanted. Eventually he fell from Robert's good graces, but seeing as Ubiquiti as a company didn't really have a ton of checks and balances, he kept his god-level access far longer than he should've.

> * Why did Ubiquti not announce they were compromised sooner? The hack started in early December, Ubiquiti noticed the compromise on Dec. 28. Ubiquiti told the market on January 11th. Is that a satisfactory turn around? Giving them some credit for the XMas break I'll say this partially understandable.

Simple. Fear of share price falling. I was constantly given this as a reason we couldn't be transparent. Not by Robert, nor where he could hear. But it was pretty much well known that the company kept shit quiet for fear of the share price dipping.

> All the AWS configuration I'm speaking of above, I would describe as Security 101.

To keep with the metaphor, Ubiquiti couldn't even get Pre-school level security in place, much less 101. I have no idea how something even more massive hasn't happened yet. Must be dumb luck.

Speaking of, by the time I left the company, the team that was handling the door entry-way systems (UniFi "Access" I guess) had been caught with numerous security issues, not the least of which was logging user credentials in plain text (not just storing, but logging, in response to authentication events). They were also based in China and subject to Chinese laws around government access, so take that how you will.

And that doesn't really even cover most of it. That year took a toll on my physical, mental, and emotional health, not to mention put a crazy strain on my marriage. I'd rather honestly forget it, but the schadenfreude of what's going on is too delicious to ignore.

[Sirened]

>Also, for some perspective, at that time Ubiquiti kept all the hardware signing keys in a private GitHub repo that every employee had read access to. And they were in plain-text. So... yeah.

This is frankly worse than any of this other news. So there's essentially zero trust associated with the code signatures since any employee, past or present, can sign a payload. Wonderful.

[_pplp]

I've since heard that the repo has been taken down and all the keys rotated, but just kinda makes you wonder how many APs and switches and cloud keys, etc are still out there using compromised keys.

Also, even though they may have had read access, not many knew it existed. But it wasn't super hard to find (I stumbled across it basically).

Oh and then there the whole metrics collection debacle, where the controller basically phoned home about the topology of every network that it managed. Even if you opted out. Opting out just meant they fuzzed your ID so any given record couldn't be linked back to PII. Which may or may not be legal, IANAL.

But either way it definitely wasn't clear that opting out meant data was still collected. Super sketchy.

[ex_ubiquiti]

> Also, even though they may have had read access, not many knew it existed. But it wasn't super hard to find (I stumbled across it basically).

We didn't have read access until Nick Sharp and his team took over GitHub permissions and gave everyone access. Wonderful security work.

> Oh and then there the whole metrics collection debacle, where the controller basically phoned home about the topology of every network that it managed. Even if you opted out. Opting out just meant they fuzzed your ID so any given record couldn't be linked back to PII. Which may or may not be legal, IANAL.

Nick Sharp was at the core of this too! He built the 'trace' system to collect all of these metrics and had all of these ideas about how to secretly collect the data in ways that would be hard for people to detect.

He pretended to be a principled person who stood for security and privacy, but whenever he saw an opportunity for political gain he abandoned all principles. He was the only person I knew at the company who was enthusiastic about collecting all of that data.

[_pplp]

Oh god don't remind me about Trace. I had to deal with the Controller side of that and it was a damn nightmare.

He basically dictated that you couldn't use any kind or repo+deployment pipeline except for what his team was building. Which wasn't actually functional for like 8 months. So we never even got a dev or staging tier to test against for months.

And then when I ended up with access to push things along, the actual apps for the trace system we're... not well implemented.

Ugh... I could bitch about this stuff for literal days but I gotta drop my kids off.

[AeroNotix]

Oh hai people who used to work at UBNT. From reading your responses here (and elsewhere) it definitely seems our paths did not cross, but the shared sense of Schadenfreude is good and strong :)

[azinman2]

This is all pretty damning. What would you use at home instead?

[giobox]

The usual answer I've found to this question is Microtik, they strike a similar balance between enterprise feature-set and more consumer level price point as Ubiquiti sort of aims for with the Unifi line.

The quality/feature set is there and the software is well designed, even if not quite as networking beginner-friendly as Unifi has become. Mikrotik's RouterOS can do much the same tasks as Unifi's management console, and can configure for auto-adoption of APs/other hardware in the Mikrotik range just like Unifi does for their own hardware.

Most competitors (I see Aruba suggested) are priced much more into the enterprise/business buyer realm. Unifi has generally been keenly priced in this market, their latest Wifi 6 APs are just 99 dollars each (when in stock of course...). Microtik's pricing is generally comparable or cheaper than Unifi in my experience.

https://mikrotik.com/

[arglebargle123]

Secondhand Ruckus APs are a pretty decent alternative, you'll have a hard time getting AX gear for a reasonable price though.

edit: Secondhand Ruckus/Brocade switches are solid, at least on the 7000 series the evaluation key has no time limit so you're not license-limited in what you can do with them. Switches are mostly <$250 on eBay if you're buying an ICX7150, ICX7250, etc. Yes, that includes PoE models.

[pbronez]

That’s the real question. Aruba has some relevant offerings, but UniFi is still pretty unique in what it offers to prosumers.

[azinman2]

Seems like a market opportunity!

Is it just software that's UniFi's weakness? Anything wrong with the hardware itself? I've had quite good luck with UniFi in my home myself but perhaps I'm not using all the features...

[bigyellow]

> Ubiquiti kept all the hardware signing keys in a private GitHub repo that every employee had read access to

This right here is why I'll never use Ubiquiti gear. These devices are so obviously backdoored and like swiss cheese, they offer the complete opposite of security. Thanks for sharing the true facts.

[ex_ubiquiti]

> For reference, I spent a year (mid-2018 to mid-2019) running the UniFi Network team and worked with Nick during that time.

Nick's whole strategy was to find a problem, exaggerate it as much as he could get away with, and then offer himself as the hero who would fix it all.

He exaggerated or lied about everything he wanted to use for political advantage, right up to the end where he fabricated a hack and used Krebs to exaggerate it as much as possible for his own personal profit.

You have to realize he did the same thing during his time at Ubiquiti: Found problems he could use for political advantage, exaggerated them as much as he could get away with, and then amplified his lies until they were gospel. A lot of what you're saying has some roots in truth, but I can tell you have the exaggerated Nick Sharp version of events.

> There was Robert.... and then nobody knows. I asked repeatedly why we didn't have a CTO, or a COO, or a CFO, or CMO or ANYTHING and I got nothing but shrugs and "idunno" as a response for the whole year I was there.

This wasn't some big mystery. Everyone knew that Robert ran everything as CEO and the legal, marketing, and other teams operated out of the New York office.

> Nick came in and started putting "proper" AWS structure and security in place, primarily by scaring Robert (the CEO) into giving him the keys to the castle

Nick was hired specifically to run AWS. That was his job from the beginning. The old cloud team quit and Nick was recruited from his job at Amazon because supposedly he was an AWS expert.

The incident where he scared the CEO was the first of his political games to exaggerate or fabricate security incidents for political gain.

> So why wasn't anybody else notified? Simple. Because he was basically "god". If anybody was gonna be notified, it would've been Nick. He was the top of the totem pole company-wide when it came to AWS.

Yes, this. All of these news stories are missing the point that Nick was the cloud lead. You don't have to believe anonymous commenters. His LinkedIn profile will confirm it. He was recruited out of Amazon to lead the cloud efforts, but he was in over his head and had severe personal issues.

> at that time Ubiquiti kept all the hardware signing keys in a private GitHub repo that every employee had read access to.

This is another Nick exaggeration. It's true that older devices had hardware signing keys stored in a Git repo before the system was updated and keys rotated. However, those old keys were only accessible by a few people until Nick and his team took over GitHub and restructured permissions with the web portal they built themselves. In the process they made too many repos accessible to too many people.

> To keep with the metaphor, Ubiquiti couldn't even get Pre-school level security in place, much less 101. I have no idea how something even more massive hasn't happened yet. Must be dumb luck.

Ubiquiti's overall structure is far from perfect, but you were only there during the Nick Sharp era. Ubiquiti had a lot of people who took security and proper practices very seriously before Nick Sharp took over everything, but it was also a distributed company with a lot of isolated divisions. Nick Sharp got into power by taking the worst and oldest parts of the company and convincing people that everything was equally bad and that only he could fix it. If you got your security information from Nick Sharp, you'd think that Nick is the only person who can do anything properly at the company.

> Speaking of, by the time I left the company, the team that was handling the door entry-way systems (UniFi "Access" I guess) had been caught with numerous security issues, not the least of which was logging user credentials in plain text (not just storing, but logging, in response to authentication events). They were also based in China and subject to Chinese laws around government access, so take that how you will.

I also heard that, but I think it was just incompetence on their part. Nick was pushing the conspiracy that they were doing something with the Chinese government, but it doesn't follow that they'd do it by sending the data to AWS servers under his control. I think they just made a sloppy prototype to impress the CEO and got caught doing dumb stuff. I do blame the company for not cutting that team off, though. They had no idea what they were doing other than their ability to put together quick prototypes to impress the CEO.

[_pplp]

I appreciate the perspective.

If you're telling me I worked there at literally the worst possible time frame, I'd believe it. I may have my experience skewed through the perspective of Nick's influence, but tbh many of my issues were unrelated to him or his sphere of influence.

The C level thing may not have been a "big" mystery, but it was to me, and as somebody who was running the dev of a flagship software product (UniFi) it set off alarm bells that nobody I talked to could explain who was handling the roles of those execs. I'm not exaggerating when I say I effectively got "I dunno" as a response when I inquired, and I dug.

It is good to know, though, that what I experienced wasn't chronic for the entire company's existence.

To clarify on the China thing, I wasn't trying to imply that anything nefarious was actually happening. Just that it warranted some scrutiny when a security focused product was being developed on the Chinese mainland and by a team of Chinese citizens that are subject to CCP laws. Given some of the things that have happened around that country's involvement in tech in recent years, I don't think such scrutiny is unwarranted, especially when the team has a track record of security "goofs".

[LilBytes]

Incredible insight, thank you for sharing.

[accountofme]

Wow.