[phs318u]

What a great article. Very easy to follow. The best part was that instead of attacking the messenger and denying any problem, Cox seem to have acted like the very model of responsible security response in this kind of situation. I'd love to read a follow up on what the bug was that intermittently permitted unauthorised access to the APIs. It's the kind of error that could easily be missed by superficial testing or depending on the reason behind the bug, perhaps not even experienced in the test environment.

[oasisbob]

> Cox seem to have acted like the very model of responsible security response in this kind of situation

It's hard to imagine, but I wish they would have taken advantage of him walking in with the compromised device in the first place.

I once stumbled upon a really bad vulnerability in a traditional telco provider, and the amount of work it took to get them to pay attention when only having the front door available was staggering. Took dedicated attempts over about a week to get in touch with the right people - their support org was completely ineffective at escalating the issue.

Cox's support organization was presented with a compromised device being handed to them by an infosec professional, and they couldn't handle it effectively at all.

[vlan0]

>Cox's support organization was presented with a compromised device being handed to them by an infosec professional, and they couldn't handle it effectively at all.

I can't really blame them. The number of customers able to qualify that a device has actually been hacked is nearly zero. But do you know how many naive users out there that will call/visit because they think they've been hacked? It's unfortunately larger than the former. And that'll cost the business money. When 99.9% of those cases, the user is wrong. They have not been hacked. I say this as someone who supported home users in the 2000s. Home users that often think they'd been "hacked".

[oxide]

I work for a support org for a traditional telco. We have "contacts" but they're effectively middlemen.

If you dropped this in my lap, and I'm pretty savvy for a layman, I wouldn't know how to get past my single channel. I think it would require convincing the gatekeeper.

[protocolture]

I used to run a small telco noc and if any of my guys sat on something like this rather than reporting it to me I would have turfed them.

Especially because both of the ISPs we supported insisted on using a lot of dodgy CPE.

[chemmail]

They probably get someone in asking to change it because someone on LOL say they just hacked their computer.

[hanniabu]

How many of those show up in person though?

[hackmiester]

Just the craziest, wrongest ones

[Shared404]

+1 to this. Dealt with the same in consumer PC repair.

[doubled112]

This was my experience too.

Some people truly believe the computer is hacked every time there is behaviour they didn't expect. Only the craziest, least capable ones show up to scream at you like you caused the whole thing.

[hollander]

That is the problem. He should have contacted them like he did the second time. When he went into their shop, it all depended on that particular employee, and you can't blame that person for not recognizing the issue.

[someguydave]

yeah the false positive problem is huge here. For every legitimate security professional there are probably 10-100 schizos who believe they are “hacked”

[marshray]

I was mentioned in the media once for an unrelated internet protocol vulnerability and I had people contacting me about their "hacked" internet connections.

For a major cable ISP, I can't imagine how many customers walk in to replace their "hacked" boxes on a daily basis.

[tw04]

> Cox's support organization was presented with a compromised device being handed to them by an infosec professional, and they couldn't handle it effectively at all.

He probably should have gone the responsible disclosure route with the modem too. Do you really expect a minimum wage front desk worker to be able to determine what’s a potential major security flaw, and what’s a random idiot who thinks his modem is broken because “modern warfare is slow”?

[VBprogrammer]

> He probably should have gone the responsible disclosure route with the modem too

I think he was probably keen to get back on the Internet to be fair.

[tw04]

He wasn’t off the internet. He just determined his modem was hacked. Given it had been hacked for who knows how long, what’s one more day? They responded to his api submission in 6 hours.

[oasisbob]

I would expect a front-desk worker to be trained to escalate issues within the org, and supported in doing so.

[thrwaway1985882]

Have you ever worked as a front-line support agent? I'm guessing not. I have many years ago, and for an ISP too. If I bought an Amazon share back then for every time a customer called support because they were "hacked", I'd not be posting here during a boring meeting because I'd own my own private island.

The two best conversations I can recall were when we changed a customer's email address about a half dozen times over a year because "hackers were getting in and sending them emails" (internal customer note: stop signing up for porn sites), and a customer's computer could barely browse the web because they were running about 5 software firewalls because they were "under surveillance by the NSA" (internal customer note: schizophrenia).

The expected value of processing requests like this any way other than patting the reporter on their head and assuring them the company will research it, then sending them along their way with a new device while chucking the old one in the "reflash" pile isn't just zero, it's sharply negative.

The author's mistake was not posting somewhere like NANOG or Full-Disclosure with a detailed write-up. The right circles would've seen it, the detailed write-up would've revealed that the author wasn't an idiot or paranoid, and the popped device might've been researched.

[indymike]

> The author's mistake was not posting somewhere like NANOG or Full-Disclosure with a detailed write-up.

This is an organizational equivalent of a code smell. Something is off when support people aren't writing up the anomalies and escalating them.

Some of the most serious security issues I've ever had to deal with started with either a sales rep getting a call or a very humble ticket with a level one escalating it up. Problem is for every serious security issue that gets written up, forty-two or so end up getting ignored because the support agent is evaluated on tickets per hour or some other metric that incentivizes ignoring anything that can't be closed by sending a knowledge base article.

[thrwaway1985882]

> Something is off when support people aren't writing up the anomalies and escalating them.

What is described in the article is a fantastic hack. Given my organization's structure and skills, you'd need to send it straight past three layers of support and several layers of engineering before you find someone who'd be able to assemble a team to analyze the claims. We'd spend four figures an hour just to confirm the problem actually exists - then we'd all go "oh shit, how do we get in touch with the FBI, because this is miles above our paygrade."

An average cable internet user walks into a retail ISP location, sets a cable modem on the counter, and says "this is hacked". What is the probability you'd assign to them being correct? How much of your budget are you willing to spend to prove/disprove their theory? How often are you willing to spend that - remembering Cox has 3.5 million subscribers.

Friction is good. Hell, it's underrated! Introduce it to filter out fantastic claims: the stupid and paranoid are ignored quickly, leaving the ones that make it through as more likely to be real.

[leptons]

"Code smell" as a programming term is often a red herring that causes conflicts within development teams (I've seen this happen too many times), because anyone can call anything they don't like about a coworkers code as a "code smell". Your comment is a "code smell". See how easy that was?

And "code smell" doesn't apply in a similar or metaphorical way towards cable modem support personnel. Those people aren't supposed to know how to escalate a case of a customer bringing in suspected hacked modem. If they did that for every idiot customer that brought in a "suspicious" modem, the company's tech support staff wouldn't be able to get anything done. 99.999999999999% of the cases would not in fact be a hacked modem, so there really shouldn't be any pathway to escalate this as a serious issue.

[Blackthorn]

You can tell exactly from the responses in this thread who has dealt with the general public in a support role, and who hasn't.

[kalleboo]

I haven't even dealt with the general public in a support role but I have enough examples just in my, not very large, social circle.

The aunt who is convinced she has a stalker who is hacking all her devices and moving icons around and renaming files to mess with her (watching her use the computer, she has trouble with clicking/double-clicking and brushing up against the mouse/trackpad. call her out on it, she says she didn't do it)

The coworker who was a college football player, who now has TBI-induced paranoia. He was changing his passwords about 3 times a day. Last thing I heard about him before he got cut out of my social circle was he got in a car accident because he was changing his password while he was driving.

Meanwhile I know zero people who have found any real vulnerabilities.

[jfyi]

I have escalated customer security issues while working as a support agent. I have also found and been paid what could be considered a bounty (in the form of a bet made by the lead dev to another person) while working support.

Admittedly, this is anecdotal, and it was a small company, and my skillset was being very underutilized at the time. However, I don't think it's hard to imagine a me that would have been closed minded enough to normalize my experiences and expect it of others. In fact, I'd say I still fight with it regardless of having seen it.

[internetter]

If the man wanted the router back, they should'a given the router back.

[Lammy]

> they were "under surveillance by the NSA"

Where's the lie? We all are.

[mandevil]

Every third person who comes in has their router hacked, that's the problem. We know that Sam is good at what he does and to not be wrong about this, but Cox can't rely on everyone being that good, nor on their very poorly paid front-desk worker to have the ability to tell if they are an idiot or a expert.

Source: was a volunteer front-desk person at a museum. Spent a lot of my life dealing with people. They were sure of incorrect things all the time and could not be relied on to know.

In retrospect, Sam should definitely have hit the responsible disclosure page (if such a thing even existed in 2021) but I don't fault anyone for the choices they made here.

[worewood]

We really need to work on this definition of "expect". It's expected from them to have such training but we know that in practice that is not what happens. So we "expect" they to be trained, but what we "expect" will happen in practice is very different.

[flutas]

> the amount of work it took to get them to pay attention when only having the front door available was staggering.

I've seen this across most companies I've tried reporting stuff to, two examples.

Sniffies (NSFW - gay hookup site) was at one point blasting their internal models out over a websocket, this included IP, private photos, salt + password [not plaintext], reports (who reported you, their message, etc), internal data such as your ISP and push notification certs for sending browser notifications. First line support dismissed it. Emails to higher ups got it taken care of in < 24 hours.

Funimation back in ~2019(?) was using Demandware for their shop, and left the API for it basically wide open, allowing you to query orders (with no info required) getting Last 4 of CC, address, email, etc for every order. Again frontline support dismissed it. This one took messaging the CTO over linkedin to get it resolved < a week (thanksgiving week at that).

[LeifCarrotson]

> Took dedicated attempts over about a week to get in touch with the right people - their support org was completely ineffective at escalating the issue.

Sounds to me like their support org was reasonably effective at their real job, which is keeping the crazies away from the engineers.

It's even harder for me to imagine them saying "Oh, gee, thanks for discovering that! Please walk right into the office, our firmware developer Greg is hard at work on the next-gen router but you can interrupt him."

[buildbuildbuild]

I’ve often wished I could show an “I know what I’m doing” badge to support to guarantee escalation.

“I’m a three star infosec General, if I’m contacting you it’s not to waste your time.”

[sgarland]

I have a cloned key from a spare modem that I use with my router (Unifi) to allow it to connect directly to the ONT, minimizing devices in my rack.

I’ve found that this usually confuses first line support enough that they’ll listen to me if I need them to do some specific action.

To be clear, I’m not stealing internet access or anything of the sort. I didn’t want a useless modem / AP that I’d end up bridging anyway, so I extracted a key from another one, and my router uses it to auth with my ISP.

[brewdad]

It quickly becomes the Service Animal problem. When no one can or is allowed to verify your infosec credentials, everyone becomes a three star infosec General with a simple purchase from Amazon/Alibaba.

[mook]

> Cox's support organization was presented with a compromised device being handed to them by an infosec professional, and they couldn't handle it effectively at all.

They were presented with some random person who wanted to get a new modern on their rental but also keep the old one, for free. They had no way of knowing if they were an actual security professional.

[p0seidon]

Totally agree, an easy read and a great reaction by Cox. I also like that the discovery and the bug itself were not communicated in a negative or condescending way, which is sometimes the case.

[senectus1]

agreed, lets hope they dont bloody sue him into the ground for "hacking"

Its stuff like this that company's should REWARD people for finding.

[acdha]

They have a pretty good looking responsible disclosure program which I’m assuming he checked first - it’d be surprising for someone who works in the field not to have that same concern:

https://www.cox.com/aboutus/policies/cox-security-responsibl...

[imadr]

I assumed they offered a bounty for bug disclosure? You mean to tell me that an internet provider with 11 billion in revenue can't pay someone that found a bug impacting all their clients?

Frankly he could have just sold the vulnerability to the highest bidder

[teruakohatu]

They do not:

> Cox does not offer a bounty program or provide compensation in exchange for security vulnerability submissions.

https://www.cox.com/aboutus/policies/cox-security-responsibl...

[tetha]

Mh, we have a similar thing on our website at work, but people who found serious issues still got compensated.

One big reason to put this out there: Otherwise you get so many drive-by disclosures. Throw ZAP at the domain, copy all of the low and informational topics into a mail at security@domain and ask for a hundred bucks. Just sifting through that nonsense eventually takes up significant time. If you can just answer that with a link to this statement it becomes easier.

It makes me a bit sad that this might scare off some motivated, well natured newbs poking at our API, but the spam drowned them out.

[cqqxo4zV46cp]

Don’t frame a company not parting ways with money that they could hypothetically part ways with as being unusually egregious. That’s never how it works. Not every conversation needs overstated outrage.

[bayindirh]

> Frankly he could have just sold the vulnerability to the highest bidder

Why? Ethics aside, is everything money?

[robertlagrant]

> Why? Ethics aside, is everything money?

Ethics aside, why not? That's why we have ethics.

[mjg59]

Vendors who pay bounties often restrict public disclosure, and the professional value obtained from being able to talk about the research you do may be worth significantly more than the payout

[robertlagrant]

> Vendors who pay bounties often restrict public disclosure, and the professional value obtained from being able to talk about the research you do may be worth significantly more than the payout

Professional value that doesn't translate into money, do you mean? How do you categorise that?

[pyrale]

Ethics aside, there are many thing that move people, and it's not always money. For instance, selling the vulnerability means the author wouldn't have been able to tell their story.

[bayindirh]

Because even if I remove ethics, I can't find a reason for doing something like that.

For me, doing the right thing is beyond all these things, and I don't care about money beyond buying the necessities I need.

[WarOnPrivacy]

> I can't find a reason for doing something like that.

Money often starts out as necessity or one of it's close cousins. If I were 1) 8k miles away from my target, 2) in a region with more internet access than employment prospects and 3) needed to eat, I can see a path to profitable disclosure.

> For me, doing the right thing is beyond all these things,

This can be a luxury. After a year or 3 of kids in and out of hunger, what's right can get reframed.

> and I don't care about money beyond buying the necessities I need.

Getting beyond that is the thing.

[robertlagrant]

> For me, doing the right thing is beyond all these things

That...is ethics, no?

[imadr]

>Why?

So this security researcher can keep doing his research without worrying about paying bills. The company gets cheap security audit, the researcher gets money, everybody wins

[naikrovek]

because money grants wishes, and having more money means you get more of your wishes granted.

[bayindirh]

That doesn't make me interested. I don't get all excited about the things money can buy.

Edit: As I noted elsewhere, necessities are something else.

[WarOnPrivacy]

May I suggest a decade of red state hunger-level poverty? My kids and I did it. Three years out of it, I get excited paying utilities on time.

[johnisgood]

I would love to have the money (42k EUR) to buy Scewo, which is an advanced self balancing stair climbing wheelchair. Sadly enough, I doubt I will ever be able to.

[unclebucknasty]

>...can't pay someone that found a bug impacting all their clients?...he could have just sold the vulnerability to the highest bidder

This attitude is why "independent security researchers" offering to present unsolicited findings to companies in exchange for payment feels exactly like extortion.

[zdimension]

At the same time, Cox is a commercial entity that makes money by providing services. Cyberattacks make them lose money, so it's only fair for them to financially award people that responsibly inform them of vulnerabilities instead of easily and anonymously selling those.

We're not talking about a grandma losing her wallet with 50 bucks in it and not giving money to the guy that found it and gave her back.

[unclebucknasty]

>it's only fair for them to financially award people that responsibly inform them of vulnerabilities instead of easily and anonymously selling those.

Yes, Cox has that choice. But, what you're describing is the definition of extortion. The fact that it's easy for people to get away with it does not make it ethical.

[bongodongobob]

It's not the definition of extortion. If I walk past a business and notice the locks on their windows are rusted and I happen to be a lock guy and say hey, I noticed your locks are fucked, I'd be happy to consult for you and show you how and why they are broken, that's just doing business. Extortion is telling them, hey, your locks are fucked and I'm telling everyone unless you pay me. It requires a threat.

[pixl97]

At the end of the day your enemy has no ethics, and we share the public internet with enemies. If paying to find security flaws means it's more likely people will find your flaws rather than sell them to someone that will use them for nefarious means then it is the better bet.

[notactuallyben]

while beg bounty people can be annoying, you have to remember that people aren't obligated to sit down and find free bugs for any company (especially not a big one) - why would i sit down and look at some code for free for some giant corp when i could go to the beach instead?

[unclebucknasty]

No, they aren't obligated. So, if there's no bug bounty program in place, then they should either go to the beach or be willing to find bugs for the public good.

The idea that the company owes them anything for their unsolicited work is misguided. And, if they present the bugs for money under the implicit threat of selling the information to people who would harm the company, then it's extortion.

[depressedpanda]

1. Companies are amoral entities, and given the opportunity have few qualms about screwing people over if they can profit from it. Why do you expect people to behave ethically towards entities that most likely won't treat them ethically?

2. If said person doesn't present the bug to the company, but just goes straight to selling it to the highest bidder it's not extortion. If the company does not provide the right incentives (via e.g. bug bounties), isn't it their own fault if they get pwnd? They clearly don't value security.

[imadr]

I would agree with everything you said, If we ignore the fact that the company has billions of dollars in revenue and paying a bug bounty is a drop in the ocean for them.

Do you think it's reasonable to say the the ethics of what you call "extortion" should depend with how big the company is? I'm obviously not advocating for making a small company pay more than they can manage

[webninja]

Yeah let’s hope that they don’t prosecute him under the CFAA. He saved the FBI and untold others. He’s a hero.

[ImPostingOnHN]

> I'd love to read a follow up on what the bug was that intermittently permitted unauthorised access to the APIs

I would, too. Not sure we will ever learn. Maybe a load balancer config that inadvertently included "test" backends which didn't check authorization?

[randomcarbloke]

it's good but the constant use of "super" was a little off-putting, "super curious", "super interesting", "super interested", etc.

[armada651]

There were 4 occurrences of the word "super" in an article with more than four thousand words in it, there is no need for "etc." you quoted all the occurrences since "super curious" was used twice.

[shermantanktop]

I guess that reader is super sensitive.

[randomcarbloke]

Once was enough.

[pfdietz]

Super off-putting, you mean.

[sgerenser]

IMHO, your comment is super nitpicky.

[randomcarbloke]

You're goddamn right.