Hacker News new | ask | show | jobs
by FartyMcFarter 1304 days ago
The people claiming that Twitter still being up proves they were vastly bloated need to read the following paragraphs until they "sink in":

> Our entire account team turned over multiple times in 2 weeks. We had multiple people (AE, AM, analyst, creative specialist) supporting our account and they all vanished without so much as an email. We finally got an email with a name for an AM last week but they quit and we don’t have a new one yet.

> Ads UI is very buggy and login with SSO and 2FA broken. One of my campaign managers logged in last week and found all our paused creatives from the past 6 years had been reactivated. Campaign changes don’t save. These things cost us real money.

9 comments

That's the engineering bias at play. While I agree that can still be "up" with a skeleton crew of DevOps and SREs, being "up" doesn't mean making money. It completely ignores the fact that Twitter makes their money on B2B with advertisers and even in 2022, those are human social relations. Firing the people on the Twitter side of that equation destroys those relations and ultimately the dollars they represent.

It's perfectly natural for an engineer to de-value B2B relations - I expect it, but at the same time that's the grain of salt we should take when we read an engineer's hot takes.

It's also what Musk's fans often refer to as 'moving the goalposts.'

Keeping Twitter up is not success for Elon

He needs to make it worth well more than the $44bn he paid for it in order to achieve anything resembling success; the Twitter is going to crash thing was just a dumb hashtag, no one believed that he's just let it go down and stay down.

what would that even look like? Yet, not achieving that insane failure is now what people cite as success.

> Twitter is going to crash thing was just a dumb hashtag, no one believed that he's just let it go down and stay down.

I do think Twitter is likely to crash. A machine can continue for a while if you stop maintenance, but at some point you'll run into a problem that needs intervention, and Twitter has lost enough of the people who know how things work that there's a good chance they'll have a serious outage.

I don't remember if I read it here or somewhere else, but someone noted that if you remove all the fire extinguishers from a building, the building will keep standing, but watch out when a fire starts.
Twitter in shambles is a very attractive target for firestarters. The conflagration would be epic.
Yep, it's pretty hilarious to me that after all the grand promises made about improvements Elon and his cult have been reduced to boasting hysterically about Twitter just merely still being up and running.
I don’t get the cult thing. I encounter way more people who hate musk, often times irrationally than I do big time supporters. I’ve seen plenty of positive takes on him but not people fawning over him. Maybe it’s because I don’t use Twitter and that’s where they’re found but at least the platforms I’m on are dominated by people calling him a muskrat and stuff. Is this anyone else’s experience?
> I don’t get the cult thing. I encounter way more people who hate musk, often times irrationally than I do big time supporters

I encounter way more people who hate scientology than people who praise it, that seems consistent with the cult label.

I'd be curious to know some of the "irrational" reasons people dislike Musk, I think people's dislike of him is pretty understandable: he's a dishonest businessman and a partisan troll.

I have no doubt that most people, let alone people who have accumulated that kind of wealth have done plenty to deserve some ire. It’s the focus on him that can make it seem irrational and like it’s more because he’s a loud mouth. There are a lot more businessmen doing substantively worse things for the world that don’t catch an ounce of hate. For me that looks a little superficial and does resemble something closer to hating a pop culture icon.
> I don’t get the cult thing. I encounter way more people who hate musk, often times irrationally than I do big time supporters.

These things are not connected, it actually makes no sense to say that. I know many more Musk haters than fans but at least 2 of the fans I know are definitely into the Musk-cult where they believe everything Musk does is great, that he has no flaws and any perceived flaw is just a tactic for him to get even further. They really do subscribe to the Musk 4D-5D chess stuff.

That's a cult behaviour, they are just into the personality of Elon Musk with no regards to his actions. The successful outcomes of Musk validates, to them, anything he does, like he is an infallible demi-God.

That's the cult thing.

Really not sure why you are conflating with how many hate or love him, there's no connection to that, at all.

That’s my whole point though. That I don’t encounter those people and my only exposure to the cult is people who hate him/it talking about it. You’re even telling me it’s a much smaller group than implied by all the references to it. I don’t know what is so confusing about that?
I used to think I hate Musk, then I realised I can't stand the fanboys who keep repeating talking points without possibility of engagement(you can't reason with those) and have no problem with Musk and I actually appreciate his way of thinking and it's worth listening to.

Musk lost quite of non-fanboy supporters though, over they years he did things that not everyone can stomach. No one forgot his pedo guy accusation for the diver who saved the kids from a cave in Thailand.

It's sample bias :)

Unfortunately, it became not just politicized but partisan. Republicans 'like' him and Democrats do not.

A long time Elon hater here. It is very infuriating to see this guy do anything and people cheering him on. Everything he does is more infuriating then the last, and I can’t help hating him more and more every day. I think this is a very human feeling. I also think it is very healthy, we should hate people that hold so much power through their wealth. We can’t elect him out or anything, the only thing we can do is hate.
This sounds extremely unhealthy.
> I also think it is very healthy, we should hate people that hold so much power through their wealth.

War is peace, freedom is slavery, ignorance is strength, hate is healthy?

How is it "healthy"? Sitting there hating some random person you don't know "more and more everyday"? That sounds like someone who needs therapy or, at least, a new hobby.

Well they didn't boast about it being up until Twitter failing became a popular prediction on the left.
"on the left?" Why is everything politicized? It's not like the Democratic party is pushing for the collapse of Twitter. How is saying the moves Elon is making are risky a "left" position? Why does everything need to turn into politics? Why can't people just be criticizing the way Elon executed the acquisition? Is the "right" so obsessed with billionaires that any criticism has to come from "the left?"
But he's not the source of politicization. Hes just observing it.

For those that think Twitter is going to crash, are you on the right?

How could it not be political? The reason Musk bought twitter is because of the censorship complaints from the right.
Exactly.
Some people just cannot tolerate free speech. They have to use any method to shut it down including ad hom attacks and using labels like ‘cult’. He’s owned the platform for a few weeks. What changes do you expect are possible in that time?
>Some people just cannot tolerate free speech

Yeah, to the point that they fire employees on the spot for providing factual information.

"Freedom of speech does not exempt us from suffering consequences" -- how many times have I heard that in the last few years?
They're not boasting that it's up and running; they're boasting that the people that tweeted about Twitter burning or going down are idiots. And they are idiots.

If they're boasting about anything, it's that Twitter is getting more users, is faster, and there's less hate. Before Elon, Twitter was a place where employees were paid $180k to endlessly redesign icons and have DEI meetings. The new Twitter is a lot leaner.

The new lean twitter has no payroll department to pay wages.

https://www.businessinsider.com/twitter-payroll-finance-depa...

But Elon has also stated multiple times that Twitter is the best and most accurate place for news. This is directly contradicted by the fact that the "twitter is burning down" rumor was spread on twitter and lots of uninformed people believed it because they saw it everywhere on twitter lol.
I will openly admit that I did believe that he managed to do enough damage with his firings that the remaining engineers won't be able to keep it running because some critical teams and their institutional knowledge have been completely lost. I'm still surprised it's running so smoothly.
Based on what I've heard from former engineers there, there are still plenty of opportunities for things to go very wrong. Maybe not this week or this year, but the sheer amount of lost knowledge means they're really skating on thin ice right now.

If they were to have a seriously catastrophic incident, it's conceivable that they might not have anyone left who knows how to get things back online. It's not inconceivable that they could eventually figure it out, but without that essential knowledge, who knows how long that could take? I imagine it would be pretty bad for Twitter's future if the site were unavailable for days or weeks on end.

I'd put all my chips on George Hotz, but he's got a lot of work to do because he'll only be there for 12 weeks.
George Hotz is asking on Twitter for people to solve his problems for him, because he is incapable of doing basic web development things like removing the login pop up: https://twitter.com/corg_e/status/1595287073547862018
George Hotz is an amazing technical person but you are putting him in a pedestal of Saviour that... Isn't really how technical stuff works.

Twitter is a quite complex distributed system, as good as geohot is he's just a guy, with one brain, capable of holding a limited amount of information and context at a time. There's no way that one guy, no matter how good, can actually completely reason about all aspects of a complex system.

Even less in 12 weeks, that's actually absurdly preposterous.

i get a million crypto dms now, thats pretty annoying
I think it's more likely Twitter gets hacked. The turnover, loss of organizational knowledge, lack of compliance department, lack of proper onboarding for new hires, etc all make a social engineering attack far more likely. Plus a plain old security vulnerability gets more likely as they try to add anything to Twitter without the knowledge they used to have of how it all works. This is likely one of the reasons the blue check has now been delayed indefinitely.
> moving the goalposts

The Internet is big and full of people.

Is it actually the same people changing their predictions as time and evidence accumulate, or is it just a change in who gets attention?

... yes? The Internet is big and full of people, both things are happening.
It seems bizarre to call this moving the goalposts. You don’t have to list every single goalpost any time you mention one. Especially when these are extremely obvious goalposts like “the site should be working” and “the ads sales tools should be working.”
But Twitter/Musk's detractors are moving the goalpost. The initial argument was that twitter would not be able to function without much of the staff that were fired, that they were somehow imperative to the site staying up. Now that it's been over a week, and the site is taking on more traffic than it has in a while, the goalpost is being moved to "this will never be profitable".

I understand people hope it will fail, because they don't want to admit how useless much of the staff these big tech companies have accumulated over the years are to the core products. But, you will only be setting yourself up for disappointment. Elon has done much more impressive things with a small team of "hardcore" engineers (as he puts it). I think many sticking around are the type not to shy away from a challenge, though again I understand many are not in a position to be playing competing in the workplace.

> The initial argument was that twitter would not be able to function without much of the staff that were fired,

You are missing out some important nuance there: no one said it would fall off the earth immediately, that it would cease to function right now. Many, myself included, have said we think it can't function long term, or even medium term, like this.

Day to day running of the main app is all automated when things are OK so it will keep ticking over as long as someone says the infrastructure invoices. The real test comes when there is next an infrastructure issue or some other fault: are the right sort of people there to resolve it quickly? Also do they have a good combination of people around to work on those bugs those advertisers are concerned about and other maintainence & improvement (of both the app and the other infrastructure it and the company relys upon)?

If we get rid of all car mechanics your car won't break down immediately, but good look getting it sorted easily when it eventually does develop a fault.

Twitter was somewhat bloated, I agree there. But what has happened to it in recent weeks is far more damaging than that could ever have been. It needs to turn around very quickly to survive financially and technically, and I think the chances of that happening are small.

> no one said it would fall off the earth immediately

This is very disingenuous, there were hordes of people claiming it would be down in 24 hours, 48 hours, after the weekend etc. and there are still masses of people still saying it won’t last two weeks. At this point so many people have cried wolf loudly and repeatedly that I think the best option is to believe no one unless they are reporting first hand facts that can be independently verified. Everything else is just hot air.

And we know that social media amplifies extreme voices.
> > > The initial argument was that twitter would not be able to function without much of the staff that were fired,

> > You are missing out some important nuance there: no one said it would fall off the earth immediately,

> This is very disingenuous,

Fair, I matched hyperbole (the implication that a majority were saying something) with hyperbole (staying no one was).

Let's go with no one with a clue who is unbiased by direct connection.

> there were hordes of people claiming it would be down in 24 hours, 48 hours, after the weekend etc

If this pot may comment on the a kettle's underside for a moment: "hordes" may be as disingenuous "no one". It was said by some angrily¹ on their way out and repeated and amplified by the mob that is Twitter² users.

And that outgoing team didn't say it would fall down in any time frame IIRC: just that things could not run without them. I would read this as a medium/long term view, the social media amplifier read it as "it dies in 3, 2, ...".

----

[1] rightly so, but the emotion does reduce the ability to maintain objective reasoning

[2] one of the selection of reasons I have for not using Twitter aside from occasionally looking at a message linked elsewhere: it is too full of certain types who think Twitter is a good idea!

Yes, and you can easily find these tweets right now with a search. People claiming it would crash and burn within a week or less.
"no one said it would fall off the earth immediately, that it would cease to function right now"

Quite a few people said that it would crash essentially immediately.

You can find “quite a few people” to say literally anything.
> The real test comes when there is next an infrastructure issue or some other fault

My money is on certs for critical services expiring.

My money is a critical hardware failure somewhere. Certs are known to be an issue and many companies have them. Twitters hardware is probably far more bespoke.
Mine is on some resource leaks. Some team knows about it and they're all fired/left now.
As the sibiling comments have already stated, that nuance was entirely absent in the claims people were making.

Twitter is a glorified message board. You don't need more than 50 engineers to oversee that. Wasn't instagram and whatsapp overseeing >250m DAU with around that many engineers before their acquisitions?

You don't need more than 50 engineers to run a glorified message board, but you sure as hell do to run the ad spend management front-ends that actually pay the bills.

Twitter-as-140-characters-shouted-into-the-ether can be 'built in a weekend'. Twitter-as-a-business-that-makes-money is an entirely different machine.

WhatsApp was point to point personal communications at that point. It's a lot easier than a broadcast system with discovery.
Having millions of DAU is potentially hard, but not the most manpower intensive part.

The hard part is monetizing.

What was the revenue Instagram and Whatsapp were collecting at the time of their acquisition? Likely 0.

The moment you start collecting money, signing contracts, besides the non engineering work like legal and finance, not to mention sales, you will need to do a lot of tracking whether you delivered what you claimed you sold.

Didnt Facebook get in trouble for exaggerating ad metrics?

And that's on top of all sorts regional compliance requirements over GDPR, cookies, which require both engineers building stuff as specified by legal, which wasnt required before because if you have no revenue, you dont really need to worry about getting sued or banned in the region.

For perspective, the staff in 2018 was 3700. Now maybe 2700.

Of course, if you were vital to Twitter, then you likely left before he took over.

Because you could. Plenty of people would rather have severance.

I base this on experience. I was a consultant so I did what all good consultants do when holding together a sinking ship: I doubled my rates. Twice.

They had ~2900 before the ultimatum with severance. I'd guess they have less than 2k now if not 1k
Staffing count != properly staffed
Absolutely. Also even if the staffing count is good, the mix between departments may be bad.
And suddenly everyone is extremely concerned about the state of Twitter's finances.

Hating on Twitter has become just another political in-group signal. Yawn.

I hope the outrageous irony of this comment is not lost on you.
> The initial argument was that twitter would not be able to function without much of the staff that were fired, that they were somehow imperative to the site staying up.

Again, what does this even look like? Twitter.com returns 404 and everyone gives up?

At least make straw man arguments that are reasonable.

Two separate things there, just as you point out: being "up" and being profitable.

But my original criticism was about all of the hysteria in the last two weeks about Twitter "going down". Those sky-is-falling tweets got very little pushback. The claim was not about profitability or advertising; there were only these shrill voices claiming the site would stop working. And when that didn't materialize, all the people making those claims just went about their business without any personal repercussions or call outs. It was all just motivated by irrational Musk hatred generated by politics. It had nothing to do with technology. You can still see these tweets right now, a lot of people didn't delete them when Twitter didn't burn.

I always held out the possibility that Twitter could die, but if it does it will be a slow death. It all depends on if the new payment model generates enough cash to offset advertising losses. So in that sense I agree with her conclusion.

What makes someone's online voice "shrill" as compared to yours?
Things like emotion-based statements. Excessive exclamation points. Complaints that have no basis in fact and no proof. Classic example: "Twitter is burning LMAO!!!"
And likewise, it's easy to keep a service "Up" if you don't deploy any code changes. Unfortunately, feature launches, bug fixes, one-off backfills, and all the other nonsense that comes with actually running a business tends to require deploying code.

You can fire everyone and fully re-staff the firm, and eventually get to the point where you can do all of those things safely. But it's going to take a hell of a long time and downtime to shake out all the things that can go wrong in the myriad of unplanned interactions between different services.

I don't expect Twitter to crash and burn in the sense of the website being down and not coming back up. But I do expect feature velocity to grind to a halt, until all of these self-inflicted wounds are staunched.

> with a skeleton crew of DevOps and SREs

Except didn't he fire them too?

He still has a few hundred people who couldnt afford to lose their job due to H1B status, and probably a couple dozen people who are members of the cult of Musk.
This is so damning. Twitter always had mediocre ROI as an ad platform in my experience. Now it is literally becoming dangerous for your business. Imagine a 10-100k per day ad campaign from years ago being made live by buggy software and not realizing it...
Wouldn’t you have standing to sue them if they did this? I guess it depends on the legal agreement you have and how it stipulates they can bill you
There was definitely a bug and everyone knows. No one is going to pay for the accidentally reactivated campaigns, and if Twitter tries to force the matter it'll turn into a bunch of expensive individual suits plus at least one expensive class action for them.
suing them only helps if you think they'll still be running in a year or two.
Twitter advertiser here. Even before the takeover Twitter advertising was broken and unreliable - starting a new campaign on a desktop would fail as the payment UI loaded a bunch of internal DNS names and failed.

Note again: this is a few weeks BEFORE Elon took over. So yes its broken, I think they generally weren’t working particularly hard on it before either.

WTAF? At the end of the day, advertisers are the real customers. How is their experience not a priority?!
IDK from what I hear many Twitter employees basically worked on anything they felt like. At Google advertising rules the roost and everyone knows it.
>> One of my campaign managers logged in last week and found all our paused creatives from the past 6 years had been reactivated.

This happened to me as well. Absolutely wild and will go to my grave before paying a dime.

This is likely temporary and advertisers have a short memory. If the platform starts targeting people more effectively and CPMs / CPCs are decent, they will be back. Most of this should be self-serve with third-party marketing agencies helping clients anyway.
You don’t seem to understand WHY they are leaving. This isn’t some moral solidarity thing that they are doing for PR reasons.
> Ads UI is very buggy and login with SSO and 2FA broken.

The rest of Twitter is pretty solid. Why this? This is the money maker. Is it because product wanted the human touch of sales instead of self-serve?

There's a bunch of bugs. I got to reload more often than previously as tweets don't load. In the DMs view it tells me all my contacts joined Twitter in 1970. Bunch more of those small things. The core works, but there is a notable degration. I also assume that can be stabilized with acceptable (to Musk) effort and then it loves on, while they rebuild knowledge about old and work on something new (be it new features or a new replacement system)

There are discussions around a WeChat-like "everything app" I can speculate that one of the "success" paths is to have that "everything" is somewhat built around Tesla, which is quite a pivot, but hey, don't you need a Tesla Assistant communicating with your friends and providing your news during the commute and for paying for coffee while AutoPilot guides you thought the boring tunnel!? And then selling it to Tesla and thus offloading some of his debt to Tesla shareholders (see SolarWorld)

Sure, will take a bit, especially so that SEC and other Tesla shareholders won't go after him, but an independent IPO seems far off.

> all my contacts joined Twitter in 1970

I noticed that as well, was wondering if it was already there or not as I am not avid user of the DMs.

The rest of Twitter is not solid. I’ve seen more bugs in the last two weeks than in the past two years.

Caveat: the Twitter nonsense has me using Twitter more than before. It’s like Cubhouse a year ago.

> Twitter nonsense has me using Twitter more than before. It’s like Cubhouse a year ago.

And it's mostly news about Twitter itself. It's hard to scroll without having people talk about Elon Musk. Had to unfollow him and still getting Elon related tweets, it's boring to say the least.

Yes! I used to do lots of new follows which has turned into lots of unfollows. Particularly Twitter blue people.
All speculation:

>login with SSO and 2FA broken.

I believe this was the result of turning off the "get rid of all the microservices " directive Elon pushed. I don't think it was malicious; some badly name microservice that was probably in charge of sending text messages or generating session tokens was shut off and the dependency chain wasn't fully understood.

>AdsUI

Anyone spending enough money for Twitter to case likely has an account manager who does everything for them. Anything that isn't in the "happy path" of the AdsUI probably gets handled by some engineer making some API calls to a prod API because it's unique enough to not worry about putting AdsUI and urgent enough where someone had to take care of it now.

> Anything that isn't in the "happy path" of the AdsUI probably gets handled by some engineer making some API calls to a prod API

Prior to going private, Twitter would have had recurring Sarbanes-Oxley audits. Auditors understand the need for occasional emergency break-glass methods of making manual database queries or API calls, but they are less tolerant about that being a normal way of operating.

Plus, if you use emergency access often you'll eventually waste more time explaining each individual access to auditors at the end of the quarter than it would have taken to just implement a UI for the feature in a code-reviewed and audited internal admin console or user-facing UI.

> Anyone spending enough money for Twitter to case likely has an account manager who does everything for them.

Did you miss the part where they are cycling through AMs several times a week? ("We had multiple people (AE, AM, analyst, creative specialist) supporting our account and they all vanished without so much as an email. We finally got an email with a name for an AM last week but they quit and we don’t have a new one yet.")

> Anything that isn't in the "happy path" of the AdsUI probably gets handled by some engineer making some API calls to a prod API because it's unique enough to not worry about putting AdsUI and urgent enough where someone had to take care of it now.

LOL who? They can't even hold on to undifferentiated sales people for more than a week at a time.

I’ve been noticing bugs, particularly around not having tweets load, and for a few days it seemed that iMessage previews were quite broken.
Customers like Coke or Nike don't want self service though.
Coke and Nike don't directly deal with twitter tho. Agencies deal with all the delivery of adverts to the different advertising platforms. The agencies use what ever is available. They would sit and make dozens of different sized video files to fit different adverters specifications to upload into Facebook, Youtube, Twitter, etc.
That's only when the budgets are small that they pawn off the execution to some operations intern. For large budgets you have every chef in the kitchen on calls with the Twitter's Ad Sales manager who in turn will make sure all the assets work in the system.
Because the author seems to have a bone to pick.
> One of my campaign managers logged in last week and found all our paused creatives from the past 6 years had been reactivated. Campaign changes don’t save. These things cost us real money.

That....can't be coincidental, and smells an awful lot like fraud.

Large accounts >$1mm spend routinely get zero support on platforms such as Facebook. That side of the business also goes through ridiculous re-orgs.

This quote makes it sound as though they had an AE/AM visit them every week ? Most advertisers count themselves lucky to hear from the ads teams once a quarter outside regular scripted emails.

That’s not my experience at all. My employer is way below $1m annual spend on FB and our in-house ad manager has regular check-in calls with a FB rep who is local to our city. Same with Google and LinkedIn.
Those teams help grow revenue. The more help they give, the more the advertiser is likely to increase their spend. They do a lot of heavy lifting to help educate ad teams about about new ad products and software changes.
>Ads UI is very buggy and login with SSO and 2FA broken

if all of those have somehow deteriorated in 1-2 weeks, this is further justification for axing the people who made them.

If Twitter goes down, it just shows that all those people were incompetent, and he should have fired them.

If Twitter stays up, it just shows that all those people were unnecessary, and he should have fired them.

2fa broke because Elon ordered all the micro services to be shut down
I've heard this claim repeated, but I've been unable to find a link to causal evidence. Do you happen to have a citation?
I've had a support ticket open since Saturday for 2FA. (Didn't try sooner than that, but the timing of when it stopped working suggests it's possibly related to Elon's tweet.) No response on my support ticket and it still doesn't work.
Thanks. I’ve seen folks report problems with 2FA (which since resolved). Correlative evidence is indeed evidence, but yet it is still not as strong as causal evidence.
If I've got something that needs an hour of care every week by a sales person, https://xkcd.com/1205/ puts it at about 10 days (two weeks) of work to make it more efficient.

However, if shaving that 1h a week off takes more than 10 days of work, it is probably better to leave it as it is.

https://www.fincher.org/tips/General/SoftwareDevelopment/Bug...

This bug tracking works on Pain / Effort / Frequency and Risk / Effort / Verifiability

Realizing that I'm making up numbers here (and you can make them up too for the story we're telling each other):

    Pain   : 2 (low pain)
    Effort : 2 (low effort)
    Freq.  : 4 (fairly often)
We've got a PEF of 8 there. Note that this is the pain for the hypothetical sales person - not the pain for the end advertiser who the sales person works with.

So, what's the risk of making the change to make it so that the sales person doesn't have to do that 1h/week task? How much effort does it take? How easy is it to verify that its actually working?

And so now we've got bugs where we can put them on a grid and say "these fixes are the low hanging fruit" while "these other ones are risky and don't get us much."

Its quite possible that that this issue would never get into the "this is painful and so we need to do the analysis for it" or "this is low hanging fruit that can be fixed and have a big impact on 'stuff'".

Nonetheless, you've got something that takes 1h/week from someone who knows what they're doing with their accounts... and they're not working there anymore. Even in a perfect org, this could fall apart rather quickly.