[A1kmm]

> All of the AI projects we have observed as a team are failing. Every single one – we have seen 0% success in a year and a half, not only amongst projects we have been asked to participate in, but even within projects that we have observed in passing while doing totally unrelated work.

That's got to be hyperbole, which blows out their credibility. They chose to say 'AI' rather than, for example, LLM, or Transformer model, or Diffusion model. This means they are including a huge swathe of things dating back to Expert Systems in their claim.

And who hasn't seen productivity gains from more established AI technology - at least things like semantic search? Who hasn't seen diffusion models generating content in roles that might have done the work by hand before? Who hasn't seen some kind of regression algorithm (even using linear regression in a supervised context counts as AI - so you can absolutely do AI even in tools like Excel) improve operation productivity?

Even if they narrowed it to the Transformer model LLMs which re-ignited recent public interest in AI, less ambitious projects to give them to engineering staff to automate easy but boring tasks in the background generally have been a success. More ambitious ones that are beyond what you'd reasonably expect the models to be able to do - for sure, those tend to fail. For most of these, the failure is predictable in advance, while some are at the boundary of what's possible, and so it is harder to predict (these are rationally genuine R&D projects).

[Aurornis]

If you read the footnote, they follow up to say they've rejected 100% of the AI projects brought to them.

Go to their home page and one of their consulting selling points is recovering struggling projects.

One of their front-page selling points is that they use "ancient techniques" from books written prior to the year 2000, because presumably everything newer than that is bad?

> For non-executive management who might be struggling to deliver things that feel beyond their control, we have ancient techniques (see: books written between 1986 and 1999) to turn your team into the envy of the organisation, and we can drop in directly to get your team the resources it needs to save a struggling project.

This is entirely a selection bias issue that they've created for themselves: Advertise a consulting service for saving failing projects to companies that don't have internal expertise to handle it, then write blog posts that 100% of the projects you see are failing. Also refuse to help them, to guarantee they can't be converted to successful projects to keep the success number at 0%.

[alexpotato]

> One of their front-page selling points is that they use "ancient techniques" from books written prior to the year 2000, because presumably everything newer than that is bad?

I get what you are saying here.

At the same time, was reading Deming's last book [0] and there is a great line that I have never seen in any recent business books:

"The organizational chart is usually a pyramid. The main benefit of this style of chart is to clarify who reports to whom. Notably absent from this chart: the customer."

I've also worked at firms filled with very smart people who had no concept (or were not incentivized) to think about the Theory of Constraints, Total Quality Management or that the firm is supposed to work as a team and not a collection of fiefdoms.

What I'm saying is: just because a book was written before 2000 doesn't make it irrelevant.

0 - https://amzn.to/4waEjA2

[Retric]

There’s some selection bias involved especially based on their marking, but you’re misreading what they are saying.

> even within projects that we have observed in passing while doing totally unrelated work.

The kind of companies with failing projects seem to be very bad at using AI. That’s different from the normal mix of success and failures at most large companies.

[Aurornis]

Companies with projects that are doing well don't try to hire consultants who specialized in saving failing projects.

No company with a good AI strategy is going to go to the Hermit Tech website where text written in Ye Olde Font explains that they'll use ancient techniques from the 80s and 90s to make your software work and thinks, "These are the right people for our AI job!"

These people are trying to carve out a niche for themselves as being anti-modern, anti-AI, and being contrarian consultants that you can bring in when you want some external consultants to agree with you in a very specific way.

[K0balt]

I think you are right on target. I’ve seen this pattern:

Company does x. X sucks, and is largely derided as being a bad idea, but it’s important-employee-bobs baby. No one wants to be the guy that tells bob that x is killing the company in some small or large way, because bob can get them fired or is protected by someone who can, so you hire consultants xyz who specialize in “transforming productivity through x management” and they come in and “transform” x into something less lovecraftian, or just explain why it’s bad in a way that makes bob look like a genius.

[Retric]

> Companies with projects that are doing well don't try to hire consultants who specialized in saving failing projects.

Large companies where every project is going well are rare.

I agree companies with good AI strategies are unlikely to use these people, but excluding failures is just as much a bias as excluding winners. AI lets you shoot your sold in the foot even faster shouldn’t be surprising, but it’s still something to consider.

[DiscourseFan]

Christ they are profiting off of reaction, they're like Trump but even worse since at least Trump is not stupid enough to be anti-technology.

[TeMPOraL]

Tale as old as time. Revolution has been merchandised. The action and the reaction, they all have suppliers, which often turn out to be the same entities supplying to both sides.

[recursive]

Check out some of Trump's positions on renewable energy.

[DiscourseFan]

Perhaps I should have specified Anti-AI. Its just the thing that upsets me the most about your everyday liberal professional, how much they hate AI just because Trump is pro AI

[teravor]

it's not surprising as LLM's will generally fail you when you yourself don't know what you want. it's surprising how many people and organizations just blunder around scribbling code without clear goals in mind - for such people LLM's are a net negative as they will just end up with even more scribbles and no hope to make sense of it all.

[BobbyTables2]

Cheaper than consultants!

[prash20026]

Supposedly 70% of software projects failed even before AI. So are AI projects better or worse than that?

Also like people have asked about the 70% failure rate, how do you define failure/success.

https://medium.com/@trienpont/why-do-over-70-of-software-pro...

[quintushoratius]

> Supposedly 70% of software projects failed even before AI.

Given the way LLMs are marketed, we should expect a vastly more favorable project success rate, not simply a close parity, and definitely not a decreased rate.

Even if we consider that the number of attempts will rise if development becomes faster, we're told to expect a greater number of projects should also succeed because they're easier. They're supposed to justify the higher costs.

It's reasonable to ask if that's the case.

[tpmoney]

> Given the way LLMs are marketed, we should expect a vastly more favorable project success rate, not simply a close parity, and definitely not a decreased rate. > Even if we consider that the number of attempts will rise if development becomes faster, we're told to expect a greater number of projects should also succeed because they're easier.

I'm not sure I follow why you would think this. Prior to AI was the reason most projects failed because generating the code was too slow or too labor intensive? That would be surprising to me if it was true.

[quintushoratius]

Generating code is not, and never was, a bottleneck to projects. Understanding the problem, designing a solution, organizing resources, testing, and customer buy-in and moving targets have always represented the real issues.

Now, with LLMs, we're supposed to be able to paper over some or all of these problems.

They're supposed to be able to understand and intuitively fill the assumptions present in vague requirements, then produce working applications. They're supposed to shrink the timelines from years and months to weeks and days. They're supposed to replace human expertise with a reliable service.

Why else would I hire an LLM service over a team of humans?

[tpmoney]

> They're supposed to be able to understand and intuitively fill the assumptions present in vague requirements, then produce working applications. They're supposed to shrink the timelines from years and months to weeks and days. They're supposed to replace human expertise with a reliable service.

But none of those sound like things that would change how likely any given project is to succeed. Shrinking timelines from years and months to weeks and days just means that you can fail faster and try again sooner (hopefully with lest costs), but nothing about that implies your rate of success is going to get better.

[s1artibartfast]

Faster and cheaper.

[zasz]

Yeah, the "ancient techniques" thing is a joke.

[TheOtherHobbes]

The article is a fun read. There's a certain amount of hyperbole and subtext which may perhaps be unfamiliar to some readers.

But the point is fair. Where are the huge AI wins? Why haven't Anthropic and OpenAI produced a regular series of case studies where Company X did Fun AI Thing and incredible success ensued?

Why do these conversations always end up mentioning chat bots?

Why do c-suites seem to be full of people chasing trends instead of leading?

Does anyone remember what customers are, or are corporations aiming for Platonic abstraction, where business activities become ends in themselves, with no useful output?

[oblio]

> Why haven't Anthropic and OpenAI produced a regular series of case studies where Company X did Fun AI Thing and incredible success ensued?

Even better, why aren't AI labs guaranteeing success if a certain formula is followed and asking for perpetual royalties for successful projects?

[rcxdude]

Yeah, and how and what someone jokes about will often tell you something about what and how they think. I don't think they'll literally just ignore everything after that point, but I do expect that they are going to tend towards an old-school approach.

[preg_match]

To be completely fair, most software projects fail, AI or not, and we've known that since before the 2000s. I'm not saying everything post-2000s is bad, but just that the core is unchanged: most software projects fail, you should expect failure, and there is no magic silver-bullet process to either anticipate or prevent failure. Agile (big A) is certainly not that process.

[dgellow]

I mean, that line regarding ancient techniques is very obviously written as satirical. I read it as „we are serious professionals, not young trend following folks“

[ElProlactin]

The prominent "Business Owners" link on this company's website is broken. "Not Found".

Perhaps this will get downvoted but I personally take with a grain of salt anything written/stated by a company that can't even get the most basic functions (like running a simple website) correct.

And the even bigger irony here is that the author has a ranty blog post in which he claims he saved his employer $500,000 by clicking a button. "[It] is fucking wild that an inefficiency that took me five minutes to solve in a GUI configuration panel was allowed to persist," he wrote.

https://ludic.mataroa.blog/blog/i-accidentally-saved-half-a-...

[fzeroracer]

I don't think this is surprising at all. I'm pretty sure I've saved my employers similar amounts in the past, because what happens is something gets configured months or years ago, the dev leaves, everyone forgets and assumes things are how they are.

I've had to delete some really silly code that would slow things down or just force waits as a result of either dealing with legacy APIs or some other arcane reason. Not without testing or making sure it wasn't there for a reason mind you, but these inefficiencies can sometimes be hidden big problems.

[ElProlactin]

The point isn't that this stuff happens. It's that a person who is trying to sell consulting services with ranty "this industry is a sham" and "it's a disgrace that it was even possible" blog posts has a website with just 5 major navigation links...and one of them is 404.

[robocat]

2 brain cells, have you tried reading below the fold of https://hermit-tech.com/business-owners

Soy un loser

[ElProlactin]

Hilarious coming from a guy who whined that he wouldn't get a $30,000 raise because he clicked a button.

https://ludic.mataroa.blog/blog/i-accidentally-saved-half-a-...

Never a good sign when a "consulting" company has time to play petulant child.

[Aurornis]

This is an interesting strategy for trying to impress potential customers.

Most consultants try to impress you by talking about the great companies they've worked for.

This blog post screams, "Look how dumb my past coworkers were!" from top to bottom, then expects us to be impressed with their experience?

[darth_avocado]

> And who hasn't seen productivity gains from more established AI technology

You have to be very careful about claiming productivity gains. There may have been some instances of gains in a specific part of a workflow but does it slow down others or result in overall gains is yet to be empirically measured and validated. We’re seeing metrics like more lines of code, better unit tests, documentation, faster PRs etc. but the actual gains of businesses are still a question mark. Do more PRs lead to faster features being shipped or does it lead to slower reviews or bug ridden code that breaks user experience? I’ve see a lot of companies tout their metrics around more code being shipped but the same companies aren’t talking about how that translates to an actual dollar amount.

[dd8601fn]

These things have been "good" for a while now. And yet, companies like Amazon and Microsoft aren't showing significant improvement in their most visible products.

If it's not obviously showing for the all-in, AI-selling companies, I simply don't expect serious improvement for everyone else.

They're undeniably neat tools, but so far there's no observable evidence that they're transformative.

[Lord-Jobo]

Amazon and Microsoft have shown considerable decay in their most visible products. Maybe it’s AI, maybe it’s just momentum, maybe it’s something else. But they AI clearly isn’t helping them and they have near infinite access(compared to me anyway), so why should I expect it to help me?

[mcv]

He does employ a lot of hyperbole. Some of his previous blog posts have titles like "I Will Fucking Dropkick You If You Use That Spreadsheet" (https://ludic.mataroa.blog/blog/i-will-fucking-dropkick-you-...) and "I Will Fucking Piledrive You If You Mention AI Again" (https://ludic.mataroa.blog/blog/i-will-fucking-piledrive-you...).

He's not a big fan of polite, corporate speech, I suspect.

But I like his style and he often has some insightful points that go against some popular industry practices.

[projektfu]

Yes, from the first link:

"This is unreasonable.

"Do I seem like a reasonable person?"

[simonw]

I think there's a big difference between individual employees using AI tools to boost their productivity - with things like Claude Code and Codex - and "AI projects" where companies build custom software on top of LLMs.

The former is easy to get right. Any software engineer (at least provided they aren't actively resisting the technology )can get useful results out of Claude Code these days.

The latter is really hard. LLMs are a strange beast to build software on, and most of the obvious projects - like the internal chatbots described in this article - are easy to have over-promise and under-deliver.

[andai]

>semantic search

I'm doing fundraising for my tf-idf startup. It's named after a very big number!

[oblio]

Outside of Google, has semantic search succeeded anywhere else?

[noodletheworld]

Look, if they have data and it says 0%, and you have vibes that say that can’t be true, who should we believe?

Do you work with lots of companies and see large AI success stories?

Or do you just vibe that you personally find AI useful so it must also be a business success?

Look, I honestly don’t care, but I think “it must be false” is also unsubstantiated hyperbole. If an agency says they see no AI success, I see no particular reason to believe they’re lying.

They’re not saying AI can’t be a success. They’re saying they haven’t seen it. That matches my experience too. Proven AI success stories seem… vague, when you dig into the details, in my personal experience.

It doesn’t seem surprising to me.

[moepstar]

...especially considering someone has to justify their token spend...

[jauntywundrkind]

Their previous article on AI shows a pretty strident ham-handedness. https://news.ycombinator.com/item?id=48002795

In general I find their submissions tend towards extreme grandiosity. I find I really appreciate people who have some nuance about the world, can see some duality, and the many many many submissions here are (I admit) often quite fun and enjoyable, but spoken much more from a bully pulpit perspective, with a zeal and self certainty that I find rarely coincides with truth-seeking.

[muvlon]

That's a completely different person, they (ludic and smileplease) just both happen to be publishing on mataroa.blog. This is a blogging website anybody can sign up for, basically Medium but for nerds.

[dannersy]

I get what you are saying, and while I'll say it is definitely not 0%, I have seen very little in the way of useful software that is primarily generated. The vast majority just does not go the distance for whatever reason. I could explain many reasons, but I am getting really tired of explaining myself. If the tools were as great as everyone says, we'd be going through a software Renaissance, but we're not. I would argue a software dark ages since it feels like things are getting worse and I find bugs in what was historically very long running and stable applications. But, whatever. I think the author is clearly talking about modern AI, I don't think they need to be explicit about models.

[j45]

I would love to learn about and from the details of their projects.

[calvinmorrison]

zero percent of statistics online are made up!

[fragmede]

[citation needed]

[ElProlactin]

https://hermit-tech.com/business-owners