[hinkley]

This is what happens when optimists win and the realists are cut out of the conversation.

As a taxi service, I believe I would want to know if I'm about to have a shortage of taxis in any one area of town, and I'd better only have a concentration in one area of town for an event the entire world is talking about, like a reunion tour or a championship game.

Even with the hack, the moment all of the taxis started converging on one area of town, alarms should have been going off and managers should have been asking questions. But that's not what happened, because we say yes the moment money enters the conversation, without bothering to ask what it says about you as a person if you'll do anything for money, or for that matter if the money is even real or just a trick to get our attention.

[tjs8rj]

It’s already so hard to build a large company, you just don’t have the resources to chase super rare, low pain outcomes.

This is the first time this has happened and the total cost of it is at most a few hours revenue. They’ll likely add safeguards to prevent such a thing now, but if they ran the company preparing for every possible way things could go wrong, they’d get absolutely nothing done.

[tcgv]

Good reasoning. Hindsight bias comes to mind:

- https://en.wikipedia.org/wiki/Hindsight_bias

[heavenlyblue]

Nobody asks you to build a large company

[onion2k]

low pain outcomes

All your customers thinking your app isn't secure any more isn't "low pain".

[ornornor]

Equifax lost millions of credit files, no consequences for them.

The Us government lost the completed forms that people who want a security clearance have to fill and that lists all their hidden skeletons (they must disclose them in the form so the govt can assess the likelihood of them being successfully leveraged by an enemy) and nothing changed[0]:

> In 2018, the OPM was reportedly still vulnerable to data thefts, with 29 of the Government Accountability Office's 80 recommendations remaining unaddressed. In particular, the OPM was reportedly still using passwords that had been stolen in the breach. It also had not discontinued the practice of sharing administrative accounts between users, despite that practice having been recommended against as early as 2003.

Not to mention the breaches happening at regular interval. I’m concerned about them and even I can’t remember them.

People don’t care. It happened to many times. It’s too abstract for a lot of people just like “Facebook and gmail can read my messages, nothing to hide”. There is little to no penalty for not being secure enough/getting breached.

[0]: https://en.m.wikipedia.org/wiki/Office_of_Personnel_Manageme...

[ivan_gammel]

99% of customers won’t care, because they will only briefly see the news, this hack did not harm them, they don’t care that much about security of an app and they don’t have a good alternative.

The impact of such incidents on company reputation and revenue is often exaggerated.

[munk-a]

A few customers will have strong negative opinions "I was waiting at the airport in the rain for four hours!" but most people will indeed shrug this off. It's a much different issue than what happens when payment systems are compromised.

A lot more people care if they're informed their credit card was stolen and told to carefully watch statements for the next month - that leverages a real PITA cost on the customer.

[golergka]

Yandex had already leaked ALL data about their food delivery customers, including addresses and names. Didn't hurt them a bit since they're a monopoly. (It used to be a duopoly, but they're acquiring the only seriously competing service now).

When you're a government controlled corporation in an openly fascist state, you couldn't care less what your customers think.

[and-not-drew]

That's got nothing to do with what we're talking about.

The first comment didn't say they should have spent more time on security, it said they should have spent time creating a system to detect if too many taxis were in one spot.

I think we can all agree that security is valuable and should be prioritized, but spending time worrying about how to stop who is already in your system from sending all the cabs to the wrong place seems like a waste of time.

Hell, IF (big if) the worst thing a hacker could do once they had access YandexTaxi's servers is send a bunch of cabs to the wrong place, you could almost spin that in a positive light. "We spent so much time protecting customer data that all they could do is send our divers to the wrong place".

[bee_rider]

It is hard to make a solid argument about perceptions. Is it possible that non-technical people would perceive the ability to send all the drivers to one location as a big security problem, even though it doesn't really require any conventional security issues? Maaayybeeee. "Hacks" that intrude into the real world do have a bit of an over-inflated appearance of importance after all.

[samus]

Maybe they managed to also steal or encrypt data, and now the media attention in a sense helps the hackers claim extortion money? Since the showlights are now on that company?

[and-not-drew]

True, but going back to the original argument, if hackers did manage to steal data, that makes the idea of spending time trying to prevent all the taxis from being sent to one place even stupider. In the world where YandexTaxi had extra time to spend on something, they should have spent it on securing their data better.

[monksy]

When you build a product, your customers expect, and pay, you to be an expert and dedicated to that domain. Not some kind of fly by night scam.

[zibby8]

This is such a ridiculous take I’m having trouble understanding if it’s satire or not.

[renewiltord]

When you write a comment, you have to be responsible. Others might read it and take it seriously and your advice might lead to death and dismemberment. If you aren't willing to get insurance before commenting, don't comment. Leave it to the professionals with licences.

[monksy]

This is more of what happens when you do the least effort to build a product to make a buck. They're probably optimized for the average happy path, however flooding isn't a concern until someone gets upset.

[lbriner]

Not necessarily. Despite us armchair critics, it is also very easy to miss an attack vector when building your software. We find stuff after years that we can't believe we missed like a missing auth check.

Not that unusual at all when you are talking about 10s of 1000s of lines of code written by different people over the span of about 8 years.

[hinkley]

That’s why I favored detection at the top. I’ve worked on complex code signing apps that the blockchain people would recognize. Shit is hard. You can’t stop many things and still make money. But if you figure out what the boundaries are of the nominally running system, you can chart or earn when you start to lose the plot.

I prefer charts over alerts, because as the company grows we keep forgetting to update the alerts. But then you need people who look at the charts between other tasks or you won’t catch anything and have to go back to alerts.

[grishka]

Yandex in particular has a system where it would dynamically adjust the price to prevent that sort of thing happening. When many people want to order a taxi to the same place, it gets really expensive, really fast. Uber does that too. This normally works well, but I feel like this hack bypassed the normal ordering system entirely and just sent bogus orders straight to drivers.

[klntsky]

New customers are allowed to pay in cash to the driver, which makes it effectively anonymous. Nothing was at stake.

[munk-a]

In most areas taxi-companies use a zone-based system where cars will flag what zone they're in (rarely automatically using GPS and more often via button presses) this is an effort by the cab company to keep their vacant vehicles well distributed to keep a high response rate and increase customer turnover.

It also happens to have the side benefit that an operator watching the flagged zones would be able to see this kind of an issue happening in advance and maybe check into why every cab is suddenly bee-lining it to zone 3.

[wildzzz]

But there should still be some override that would allow for a bunch of taxis to converge at one spot. Say a sporting event just got out, there's going to be a lot of people looking to catch a ride home. If you don't want all of those customers finding another ride, the system should have no problem dispatching drivers from other zones to pickup. Having a bunch of fares popup at the same location shouldn't be a major concern and it sounds like there were no safeguards preventing every driver from being dispatched. Without just adding a limit, like no more than 50% of taxis can be dispatched to a single zone, I'm not really sure how you could prevent this from happening again. I don't know exactly how the hack happened but if someone was just able to manually spam the dispatch queue directly, the only thing you could do going forward would be to place an automated check on every addition to the queue that it's from a real user with a valid credit card and that no other requests from that user exist in the queue.

[hinkley]

It’s a distributed system right? How do you prevent saturation of a single service?

Backpressure.

Add artificial delays to the queuing time, increasing for each taxi.

[lstodd]

Maybe that was the case 10 years ago. Or 15.

Nowadays it's all automatic, there are no predefined zones - only past statistics and about zero operators.

edit: srsly, that's what Uber is all about. and YTaxi is one of Moscow Ubers.

[munk-a]

My comment is about how Royal City Taxi, Yellow Cab Vancouver and Benways in Burlington work - I have never been an Uber driver or involved with the company and can't comment on how they manage drivers.

Also, you're saying my comment is out of date but this out of date system effectively solves the issue that just occurred with YandexTaxi - so maybe if you're working on a more up-to-date system you should borrow from the out of date tactics.

[tenebrisalietum]

There are always going to be individuals that say yes the moment money enters the conversation, as long as food and housing cost money and there is the possibility of going without.

[mike_hock]

Universal Basic Income now!

[d1sxeyes]

Yandex.Taxi, like Uber (in fact, they merged with Uber in Russia), is not really a 'taxi service', they're a marketplace.

A real taxi firm would notice and stop taking new calls to the address, but Yandex.Taxi aren't really 'dispatching' taxis, they're just advertising jobs, and letting drivers respond in real time.

In fact, I'd imagine that almost none of the orders placed are reviewed in realtime, and the only indicator that anyone would have had for this to begin with would have been a higher than average number on the dashboard for 'trips requested today' - an interesting metric, but not something that I would expect to be monitored closely in real time.

I'd imagine there's a 'no show' procedure that doesn't involve human oversight, so the first couple of drivers likely arrived at the address, waited a few minutes, then coded in the no show and moved on to different jobs.

This is also likely a metric on a dashboard which would have been the second indicator - booking cancellations/no-shows/driver rejections. But again, it's an analytics metric, rather than realtime actionable business intelligence, so it's the sort of thing that gets put into weekly reports. Maybe someone would have seen it and thought 'huh, that's a bit high', but probably didn't trigger any alarms.

Eventually a curious taxi driver would start to question why there are so many taxis outside this address, and would get out of his car and chat to his colleagues. They'd identify that they'd all been asked to the same address, and probably all cancel together and drive off.

MAYBE the third indicator here would be a call from one of the drivers to customer support, letting them know about the 'system glitch' that meant multiple taxis were waiting at the same address, but it's equally possible that the drivers just moved onto their next fare without reporting any issue.

So potentially, the first time that anyone at YT realised there was a serious issue was already 10-15 minutes after the incident occurred, by which time, it's already late. On top of that, it's unlikely that they have a way to easily and effectively cancel all bookings to a particular address.

I don't have any details on the hack itself or YT's infrastructure, so it may have been very difficult to identify and cancel the fraudulent bookings en masse (e.g.: fuzzed addresses, booking times, different users, card details not stored or different card numbers used, etc.).

By the time it got escalated to any technical teams, we're already likely 30-40 minutes into the incident itself, at which point they have to analyse what is happening, trace how it happened, and identify a fix.

With the immediate nature of taxi booking (I want a taxi NOW, not in 45 minutes), it doesn't surprise me that an incident like this can occur before any technical measures can be put in place to stop or mitigate it.

[renewiltord]

Who is cutting anyone out of the conversation? You sell your product and if I care for 100% uptime, I'll pay for it. I actually don't. I can route through lots of stuff for appropriate savings and most people can.

No one wants this single pair of instances in a Tier 4 datacenter that host a single key-pair authenticated process with dual manual approval and an air-gap that dispatches one taxi (and precisely one taxi) every 30 days on a route where it can be guaranteed to hit its time prediction.

Any fool can build a bridge that stands. It takes an engineer to build a bridge that barely stands.

[hinkley]

I guarantee you there are two ex employees saying “I told you” right now.

Faster faster faster always wins because that’s what the management wants to hear. As long as their options best before the consequences stack up, they have no - and accept no - responsibility for the longevity of the company.

If you haven’t worked with any defectors than you’ve managed better than many of us, or you’re very lucky.

[renewiltord]

Sure. But it's a market. It's possible for consumers to exercise their preference. And overwhelmingly they don't care about this stuff.