[roboyoshi]

In case people are wondering what this is about: All links and images are not working on twitter right now. Seems like they accidentally included their own services in the api blocklist? Sending best of luck to the engineers. The whole twitter saga has given me a good chunk of valuable "lesson learned" over the last weeks and months.. so thanks for that.

[Tostino]

For someone who has tried to avoid hearing as much as possible about this drama (and didn't notice this was about Twitter until looking at the comments), what are the condensed "lesson learned" takeaways if you don't mind elaborating?

[CameronNemo]

Don't fire or alienate hordes of engineers with firm-specific talent. Especially for a product/service as mature as Twitter.

[JeremyNT]

I honestly think the engineering/ops problems are the least of their failures. They bled revenue not because of outages, but because wild policy swings and chaotic management style alienated some of their biggest customers.

If they had frozen features and left the existing policies in tact, I suspect we would have a dramatically different narrative about the layoffs. If brief interruptions like this are the worst that happens when you cut engineering to the bone, it's a good argument that is Twitter was indeed wildly overstaffed.

Instead, though, we have a company in crisis due to its mismanagement of other areas, so we're primed to view stuff like this through the lens of that broader failure.

[raydev]

> If brief interruptions like this are the worst that happens when you cut engineering to the bone, it's a good argument that is Twitter was indeed wildly overstaffed.

Overstaffed in order to maintain Twitter as a static service that never ships new things, sure.

I guess they've been able to ship some things that the old Twitter had already implemented and/or a/b tested. But I'm not sure those count. Meanwhile people have been paying in advance for Twitter Blue features that were promised 3 months ago.

[norgie]

I mean the core issue is poor and capricious management either way.

[MrOwnPut]

Or else your site for logged out users will go down for an hour?

What was the reason when Twitter went down pre-Elon?

I really don't see the reason for this artificial drama. Do other "X service is down" threads go this way?

People say Elon is dramatic, but this thread is honestly ridiculous and way more dramatic than anything I've seen him post.

[silentsanctuary]

The frequency and types of outages and failures is significantly more frequent now than pre-Elon. This isn't a surprise to anyone, given Elon's strategy for maintaining Twitter (or not maintaining it, as the case may be).

I don't interpret a lot of "drama" as you put it, but interest. Many observers here are in this field, and follow the "chaos engineering" discipline. Some of them use tools like "chaos monkey" that simulates a metaphorical monkey running through your server room turning off random things, to see how well your resilience systems cope. It's a rare and greatly interesting sight to such practitioners to get to see what happens when the monkey "disconnecting the more sensitive server racks" is a more literal one.

[prenoob]

Well, the fail whale was a meme way before the musk era..

[favorited]

The Fail Whale hasn't been used since 2013.

[raydev]

Yes, almost a decade ago.

[somsak2]

is there data on this? doesn't feel like outages are more common to me.

[RC_ITR]

>What was the reason when Twitter went down pre-Elon?

To use a tennis metaphor, good players minimize unforced errors and recover quickly from forced errors.

This is a very clear unforced error that could likely have been prevented by just waiting to roll out the new feature.

To extend the tennis metaphor, it would be like Serena Williams losing a set on 50 double faults. Sure, she's lost other sets before, but it would be notable for her to lose in such a unique way, even if she still went on to win the match.

That's why people are talking about this, it's a very weird way for a site to fail and it's interesting how it happened.

[ISL]

Alternatively -- don't take on debt for an acquisition incommensurate with an acquisition-target's capacity to service that debt.

Musk's cost-cutting may yet pan out from a business perspective, but it seems to be a pretty risky move.

[MrOwnPut]

The lesson is if Elon runs Twitter, an hour long downtime takes up everyone's day so we can all speculate.

Before, it was "healthy for everyone" when Twitter went down (search hn.algolia.com for pre-Elon Twitter downtime threads)

Now it's an episode of Real Housewives just like any other Elon thread on HN.

[camjohnson26]

He created that drama when he tried to gaslight the entire tech industry into thinking 90% of engineering jobs were redundant, you can’t expect anything else than for him to get dunked on when the completely predictable consequences of that play out, namely that the service will get more and more unreliable over time.

[darthrupert]

It does hurt the proud IT worker in me, the knowledge that half of us can be thrown away and almost nobody notices.

Perhaps the takeaway is that we should get our shit together. The opposite of r/antiwork.

[MrOwnPut]

No. I think the people creating drama here should take responsibility and act like professionals.

No need to pull out terms like gaslighting to justify creating more drama.

A 1 hour downtime is nothing.

[ModernMech]

> A 1 hour downtime is nothing.

If it’s nothing, why is Musk on Twitter saying the whole codebase is brittle for no reason, and needs a complete rewrite? Sounds like a pretty big deal to me.

As far as the drama comment, you’re free to disengage from this thread at any time. No need to continually interject how against the drama you are. Just leave if you find it distasteful.

[MrOwnPut]

A 1 hour down time every infrequently is nothing, but at the same time a codebase can be brittle. Are you saying the two are mutually exclusive?

Brittle code means it's hard to change, and causes issues when changed, but other than that it's stable when running.

And I have every right to point out the drama here. People here do somewhat have the right to create drama, but it is distasteful and unprofessional and I will continue to call it out.

Glad for you to admit there is drama here though :)

[roboyoshi]

- When you fire a lot of engineers, legal, security experts... things will still somehow keep working - Also some people still want to keep working at this place (or must, because of VISA etc); It probably happens all the time, just this time everyone was watching it in full motion. - I was sure this whole takeover was the end of twitter, but somehow with how many users twitter has, it just won't die. Goes to show the advantages of the mass.. probably the same applies to microsoft/google.. - The whole 8$ for a checkmark story.. which destroyed the whole trust in the checkmark basically immediately. The whole verification process (how that worked) - many people shared some insights on that and how much effort it was to fine-tune all of that with the scale of twitter. - The free API just being shut down with basically zero communication to devs. and somehow getting away with it. I guess this is a reoccuring theme in tech that a small company has to be open for developers until it's big enough to turn on them. - With the reduced engineering and the new management, it seems more errors slip into production (and incidents like the one now) - or maybe it just feels like it with all the focus on twitter right now - but anyways, you just see things breaking you normally would not expect to see.. - How people are communicating, or in this case not communicating.

I'm probably not the ideal person to write this down, there was so much stuff going on, some people probably made a whole blog-article series on this. This is just a few of the things that I'm able to remember right now.. and with everything on hn here it gives you some ideas and things to think about. Hope that Helps.

[coliveira]

Maybe mister "genius" is working his magic by annoying as many people as he can and making Twitter visibly worse before it is finally great again! So people will finally be amazed at how "genius" he really is.

[banannaise]

Seems likely that they used an allowlist rather than a blocklist for the API restriction, and left several critical services out of the allowlist.

[shadowgovt]

> Sending best of luck to the engineers.

... mostly around the hope they can jump ship before the Titanic pulls them under.

[echelon]

Why would you stay at Twitter? Even if you believe in the platform and the mission, this has to be one of the most stressful work environments in tech right now.

The only cases I can think of for staying:

- Unfortunate H-1Bs tied to the job. (We really need to make it easier for these folks to switch jobs and keep their visa!)

- A feeling one needs the job and can't easily get work elsewhere. Perhaps living paycheck to paycheck, haven't interviewed in awhile, imposter syndrome... (If you're worried you'd not be able to get hired, instead ask why you weren't fired.)

[PragmaticPulp]

> Why would you stay at Twitter? Even if you believe in the platform and the mission, this has to be one of the most stressful work environments in tech right now.

I worked for a different eccentric billionaire early in my career. It was stressful, demanding, and unpredictable as he tried to micromanage everything by himself in short bursts of 5 minutes before moving on to the next thing.

The hook was an idea that it was a famous company and he was a [not quite] famous billionaire and that we were sitting on a once in a lifetime opportunity to build our resumes. He was going to reward us greatly at some future date when everything was working well again, and we’d be able to work anywhere else we wanted after this.

Didn’t really pan out for us as everyone got burned out, the billionaire’s micromanagement and constant product churn diminished the company’s reputation, and he eventually laid most people off in favor of even cheaper foreign labor.

[lamontcg]

Probably the exact path Twitter is on.

But there'll be one or two people who get cozy with Elon and win the lottery and will write articles on how hard work always pays off (neglecting the other 99.9% of Twitter that gets fired when it all gets shipped to India).

[jgust]

> Unfortunate H-1Bs tied to the job.

Or we could reduce the hoops & cycle time for legal immigration. The US has an opportunity right now to cement itself as the leader in technology for the next 2-3 decades and stave off the impending population growth crisis if we could find a way to enable people who want to live here to live here.

[stcroixx]

Or we could force these tech companies flush with cash to pay market rate and watch salaries rise or let these companies leave the country if they can’t afford to do business here. These are the richest companies on the planet and they don’t need a labor subsidy, they just want one and are powerful enough to get their way.

[echelon]

> leave the country if they can’t afford to do business here

Be careful what you wish for.

[stcroixx]

I was in the industry before this greedy trend accelerated out of control - I know first hand how much better things were before. I’ve seen a colleague choose death over the abuse the average H1B gets subjected to. The sooner this lie ends, the better for all of us.

[fallingknife]

I'm not interested in government policy that allows companies to bring in cheap foreign labor to undercut my compensation. There are 330 million people in the us, so if companies need more engineers, they can pay more and train more.

Also, there is no population growth crisis. A decreasing population means less stress on the environment and lower housing costs. Yes, there will be economic effects, but it's hardly catastrophic. Japan is managing just fine.

[jgust]

This is a very xenophobic view on immigration, and arguably the status quo. Past performance does not equate to future results, but I think you can draw the conclusion that the immense diversity that exists in the US compared to "rest of world" is a competitive advantage. What would it mean to lean in?

[JohnBooty]

    Yes, there will be economic effects, but it's hardly 
    catastrophic. Japan is managing just fine. 

I'm absolutely unqualified to say if they're correct, but there are a lot of economists predicting disaster for countries like Russia, Japan, and (not far behind them) China who will face this kind of demographic shift. The US is in the midst of this kind of shift as Boomers age through the system, but it is said that it will be less severe (thanks to younger immigrants) and less permanent than some other countries are facing.

Of course, you can find a lot of "experts" saying anything. Economics is an area where I haven't got the chops or the hubris to tell shit from shinola.

[echelon]

> government policy that allows companies to bring in cheap foreign labor to undercut my compensation

They become Americans and rise to your level of comfort and compensation. By working here in the US, they increase the wealth, tax base, and culture of our country. Immigration is a good thing. Skilled immigration even more so.

The alternative is that they stay in their home country, and that country grows a tech industry that rivals the US. Those workers will work for even cheaper than in the US, putting an even wider delta on price and creating an incredible arbitrage opportunity for talent.

Much like the automotive industry, foreign competition will drive margin out of our comfortable tech industry that has enjoyed being peerless for decades.

We're only going to see more `TikTok`s and `Spotify`s arise.

At some point, talent won't want to come to the US anymore. That should scare you.

I say all of this as someone who wants everyone to enjoy wealth and prosperity regardless of where they live. I still want opportunity and the ability to capitalize on it to be accessible to any American that would take it. And for that to continue, we should keep growing our talent pool and increasing the scope of what we can achieve together.

[fallingknife]

I see foreign competition as an inevitability, but I don't think number of engineers is a decisive factor. China and India have enough engineers already, and have for a while, but other than Tiktok, no real competition yet. Whereas Sweden, home of Spotify as you pointed out, has very few.

Even if number of engineers is decisive, we will never stop it by immigration. China + India have more than 8x the population of the US. We would never be able to deprive them of enough engineers via immigration.

[jgust]

^ This is roughly the point I was trying to make.

[DrewADesign]

Maybe folks early in their career. It's not like the cachet has totally dissolved because they did employ a lot of top notch people. Also, when they're interviewing for their next job, even if your departure was super contentious, Twitter is a great scapegoat that nobody will question.

"It says here you were terminated for breaking into a vending machine in the lobby and distributing snacks to your colleagues while dressed as Robinhood."

"Yeah it was an act of protest because management threatened to take away our lunch breaks."

"Well... it... sounds plausible I guess."

[gedy]

(Not at Twitter) Interviewing sucks and many people have other priorities in life, like family, etc.

[Nextgrid]

Could also be that they have enough money saved up and don't actively need a job, so they're happy to ride this one out until the end and because they don't need the job they don't really get stressed or anything - if they get fired no big deal.

If they actually make it all the way to the end, maybe being the last man standing there actually does have some perks we don't yet know about?

[TechBro8615]

Perhaps some people have different preferences than you.

[masklinn]

> The only cases I can think of for staying:

You forgot the bootlickers, I'm sure not all of them have been fired yet.

[mlindner]

> - Unfortunate H-1Bs tied to the job. (We really need to make it easier for these folks to switch jobs and keep their visa!)

They can switch jobs. H-1B doesn't force you to stay at your job.

[nickstinemates]

Another reason: some people enjoy the work environment that is promised at the "New Twitter."

Clearly not for everyone given the amount of opting out and controversy.

[secondcoming]

Everything seems fine for me when I use the main website.

[orra]

Are you logged in? Twitter seems to be working normally (i.e. it refreshes) where I'm logged into the browser. But where I'm not, I get the error message everybody's seeing.

[tsimionescu]

Same for me, but accessing Twitter links from Google breaks it.

Seems to be related to a query param in my case (if I remove the ?lang=en, the page loads properly).

[almog]

Perhaps it's region specific but I get 467 errors right now for images on both mobile android and desktop web.

[aniforprez]

Definitely region specific I think. It's working mostly fine for me. Links and images are mostly fine except for the errant occasional broken one but even then it's only when opening the full image

[yk]

Doesn't work from Germany right now. (Telekom if that turned out to be important.)

[meowtimemania]

I see the issue on iOS

[mikeyouse]

No new links or images work for me from the main website. Cached stuff seems okay.