[brettgriffin]

> Despite the rush to integrate powerful new models, about 5% of AI pilot programs achieve rapid revenue acceleration; the vast majority stall, delivering little to no measurable impact on P&L.

This summer, I built two very sophisticated pieces of software. A financial ledger to power accrual accounting operations and a code generation framework that scaffolds a database from a defined data model to the frontend components and everything in between.

I used ChatGPT substantially. I'm not sure how long it would have taken without generative AI, but in reality, I would have just given up out of frustration or exhaustion. From the outside, it would appear to any domain expert that at least three other people worked on these giving the pace at which they got completed.

The completion of those two were seminal moments for me. I can't imagine how anyone, in any field of information systems, is not multiples more effective than they were five years ago. That directly affects a P&L and I can't think of anything in my career that is even remotely close to having that magnitude.

I don't know what encapsulates an AI pilot in these orgs, and I'm sure they are massively more complex than anything I've done. But to hear 95% of these efforts don't have a demonstrable effect is just wild.

[soiltype]

> From the outside, it would appear to any domain expert that at least three other people worked on these giving the pace at which they got completed.

Did several domain experts tell you this or are you making it up?

> I can't imagine how anyone, in any field of information systems, is not multiples more effective than they were five years ago.

Perhaps "they are massively more complex than anything I've done"

[brettgriffin]

> Did several domain experts tell you this or are you making it up?

It's an assertion among eight other engineers on the project with ~15 years of experience in the domain. They are domain experts. This part isn't up for debate.

[soiltype]

I'm not questioning the credentials of your coworkers - I didn't know they existed!

Just so I'm clear then, because this adds a lot of context: Nobody else worked on the 2 softwares you mentioned, but you are on a team? Are the softwares part of a business, one that's making money?

[brettgriffin]

I misinterpreted your intent. Sorry. My key takeaway from my initial comment, based on replies, is how rare it is for someone to build software for money on this forum.

Correct, they didn't directly work on these features. These integrate into a larger pieces of software, both of which directly generate revenue from these features.

[dgfitz]

> These integrate into a larger pieces of software, both of which directly generate revenue from these features.

Were these larger pieces of software _already_ generating revenue, before your features?

First dollar is the hardest one to make.

[nemomarx]

I think they mean integrating AI into the business system directly and not using it to code things. I can see that having a more neutral impact

[brettgriffin]

> Generic tools like ChatGPT excel for individuals because of their flexibility, but they stall in enterprise use since they don’t learn from or adapt to workflows, Challapally explained.

Maybe I misunderstood this, but I took this to mean that people inside enterprises are struggling using tools like ChatGPT. They do point out that perhaps the tools are being deployed in the wrong areas:

> The data also reveals a misalignment in resource allocation. More than half of generative AI budgets are devoted to sales and marketing tools, yet MIT found the biggest ROI in back-office automation—eliminating business process outsourcing, cutting external agency costs, and streamlining operations.

But I've seen some amazing automation does in sales and marketing that directly affected sales efficiency and reduced sales and marketing expenses.

[layer8]

“AI pilots” in the article refers to developing AI-based tools, not to using AI for software development. These projects have a 95% failure rate of successfully deploying the AI tool being developed into production.

Regarding use of AI in software development (which is not what the article is about), the proof of the pudding isn’t in greenfield projects, it’s in longer-term software evolution and legacy code. Few disagree that AI saves time for prototyping or creating a first MVP.

[brettgriffin]

You are correct. As I pointed out in another reply, I misinterpreted this part:

> Generic tools like ChatGPT excel for individuals because of their flexibility, but they stall in enterprise use since they don’t learn from or adapt to workflows, Challapally explained.

I didn't read the actual report (and probably wont) so I was figured "AI Pilots" _could_ (and honestly, should!) include the deployment of models to assist in any and all work (not necessarily even just coding - I just used it as an example).

[ModernMech]

> But to hear 95% of these efforts don't have a demonstrable effect is just wild.

Why tho? You used AI to make some software, but did you use AI to achieve rapid revenue acceleration?

That you used AI to build software seems tangential to whether it can increase revenues. Over the years, we've seen many technologies that didn't deliver on promises of rapidly increasing revenues despite being useful for creating software (cough OOP cough), so this new one failing to live up to expectations isn't surprising. Actually given the history of technologies that over promise and under deliver on massive hype, disappointment should be the null hypothesis.