[danpalmer]

Let's assume that AI is human level (it's not, but humour me). Let's assume that businesses can buy an off-the-shelf black box of AI that will run whatever business IT process they want to run. Customer data goes in, actions for humans to perform come out. It's basically an automated call centre full of humans.

Now something goes wrong. A customer isn't getting what they ordered. What do you do about it?

Do you just drop the customer? Well that's lost revenue, or an easy win for another company, you don't want to do that. Maybe you tell the AI to fix the customer in a particular way and hope it will do so. If it does, all it takes is having a human available to fix it (a customer support dept, even if tier 2), but if it doesn't fix it now you still need all the software you needed before to fix it.

I've built line-of-business software before, and I've over-automated things before. Being able to inspect the state of systems, and being able to manually change things in software, is incredibly valuable. This is why spreadsheets run businesses.

My bet is that companies will still mostly buy the software they were buying before, but that some of the interactions with that software that humans would have done before, AI tools will action instead. All the same CMSs, CRMs, ERPs, and other TLAs will exist, they'll just be augmented with AI. They'll probably do some more interesting things with that AI, but they'll still exist in fundamentally the same way. Companies are not going to entrust their business to a black box that sometimes says no.

[ThalesX]

> Being able to inspect the state of systems, and being able to manually change things in software, is incredibly valuable.

I was hired as a CTO for a start-up that had a shitty solution (bad performance, bad practices - such as storing passwords in plaintext and customer service would just copy the password and log in the customer's account manually).

They had a custom back-end where they were basically handcuffed to a couple of flows. They couldn't see the full state of the system. Not even a fraction of it. Everyone hated it to the point where most were using whiteboards to work.

My first move was to normalize the database and throw in an off the shelf back-end where they got access to the full state of the system (with some aspects of course conveniently configured out).

Then, based on their own whiteboard flows and how they were using the new all you can eat back-end, we implemented a number of flows that mapped to how they actually worked. This is on top of being able to manually pull any switch. From what I could tell, they loved it, both the freedom and the ease.

It was truly satisfying to see those empty whiteboards.

[jddj]

> It was truly satisfying to see those empty whiteboards.

The true measure of success of LoB apps. That, and genuinely insightful feature requests rather than workarounds

[Harmohit]

I completely agree. I do not believe that AI is a disruptive technology like computers and the internet was. It will at best be a + 25 % efficiency for business processes - which is awesome and I am all for it. But it is not a game changer that will fundamentally change how things are done.

[danpalmer]

As with all disruptive technologies. Uber was 10x better than taxis when it was being pumped full of VC cash, now it's better enough that it's my go-to choice, but not by a 2x margin. AirBnB was 10x better than hotels when it was being pumped full of cash, hadn't realised that people might trash places, and chancers hadn't filled the platform with crap properties, now it's a wash and only another option alongside hotels. Bitcoin, well, lol.

AI looks 10x better than the alternative, while it's being pumped full of VC cash and while we haven't realised all the downsides (because so far we've only seen idealised marketing promises). It'll work out to be an expensive and not much better option.

[android521]

This won’t age well. Let’s come back in a few years.

[bruce511]

Any prediction of the future is either right or wrong. Yes, some can be really wrong (we'll never need more than 640k of memory) but ultimately a prediction is just a prediction.

Personally, I think AI is in the hope cycle right now. It's new (in an industry that sees very few new things.) I can see it being a useful tool in our belt in the coming years.

But equally, I don't think it'll replace (most of) us. I think we'll use it to be more productive.

Your target market will also matter. We're B2B, and companies don't buy "software" from us, they buy solutions to problems. And that very much includes the ability to talk to a human and have them solve the problem. (The human might use AI to solve the problem)

The point is, this is a large complex future space, and I believe the statement above will age well. AI will move the needle, but at the moment it's future capabilities and effects are also largely speculation.

Or maybe I'm just jaded having seen this cycle a lot, where most fall short, some disappear, amd a few stick around.

[worldwidelies]

How so? I think he's got a point. VC Pumps things with cash, forces them to work well on the surface, and eventually it becomes a thing that isn't so great. Uber pays shit to the drivers and takes a large chunk for themselves. Airbnb is contributing to the housing crisis. What's the problem going to be with AI products 10 or 15 years from now? I think that's a valid question if you look at the history of these types of ventures.

[danpalmer]

AI may take over the world, but we probably need 2 orders of magnitude of cost reduction, plus a significant improvement in intelligence, plus significant global legislative support, for AI to meet the promises being made right now. To be clear, I do think it'll be transformative, I just think it's "internet" level transformative over a career-long time horizon.

[EFreethought]

25% efficiency gain, but you have to burn the planet to do it.

Is it worth it?

[billforsternz]

> Do you just drop the customer? Well that's lost revenue, or an easy win for another company, you don't want to do that.

Are those the only or even main reasons you don't want to "just drop the customer"?

I hope not. Is offering a good service at an attractive price, one that delivers on its promises not even a thing any more?

[Rastonbury]

The comment says that to support their main point

[re-thc]

> Do you just drop the customer? Well that's lost revenue, or an easy win for another company

Works for Google. Lots of their stuff has non-existent support even today.

At the end of the day it might cost more to acquire the customer than not.

[danpalmer]

This is a decision that companies can take, and some do (although I'd somewhat contest the google example, I've had good experiences before working here, and I'm biased of course).

I think the issue will come when companies start dropping customers that don't fit certain templates. They'll be accused of using racist or sexist AI, or something like that, and it'll be hard to defend against that because AI is a black box. With humans you can say "we have a policy against this", or "we'll retrain our employees", but that won't be possible for AI in the same way. Yes you can "retrain" an AI, but that retraining looks a lot less like a corporate education program, and a lot more like an IT change request, and surprise surprise, we're back to buying software systems.

[portaouflop]

> They'll be accused of using racist or sexist AI

I don’t see how that would be relevant from the companies perspective. Companies get accused of this and much worse all the time and it almost never has any real effect, happy to be corrected

[devnull3]

That is the reason AWS & Azure are much more successful when it comes to enterprise customers. This is where the real money is.

[bruce511]

Google drops users not customers. It's an important distinction.

Ok, they drop customers too, and even as a customer its gard to talk to a human. And that in itself is a huge barrier to entry to some possible customers.

Personally we use Google for ads. We pay them money, but don't get to talk to a human. They'd drop us if we behave in a way the algorithm doesn't like. They get away with this because there are very few alternatives.

But we don't use GCP. We use AWS, and talk to a human a couple times a year. And issues get fixed.

So yes, it works for Google, but only really in the space where they have a somewhat-monopoly.

In other spaces like Gmail or Google+, they happily drop users all the time because those are users, not customers.

[danpalmer]

FWIW, I used to use GCP and Workspace, used to talk to someone once or twice a year, had no problems with the support. I've also seen AWS drop the ball on security disclosures, so I think experiences between all of them can be pretty mixed.

[treve]

I see this argument all the time, and they always miss one key thing.

The comparison is made is 'Can AI take over X, and do as good as a job as someone who's good at X'. But I think the reality is that these products aren't coming after the ones that are doing 'great work'.

There's tons of mediocre software being written and in many cases mediocre is kind of good enough. I think it's going to empower people who _aren't_ thoughtful and they'll be more productive than ever.

So my prediction is that companies will start trusting this black box, because it's cheap and fast. I also think a quality crisis in many fields is inevitable. We're already seeing this a bunch in very obvious ways, but its way more scary when the models are better and the errors are subtle.

[lasereyes136]

> Companies are not going to entrust their business to a black box that sometimes says no.

Companies entrust their business to a black box that sometimes says no now. AI will not change that, and I agree it will get worse.

It is the irony of automation that some people still need the skills, knowledge, and abilities even more to fix problems that the automation can't fix. https://en.wikipedia.org/wiki/Ironies_of_Automation

[wruza]

Let's assume that AI is human level … Now something goes wrong. A customer isn't getting what they ordered. What do you do about it?

Seems like you’re also assuming some AI-inherent problem which is left unexplained.

If it’s a human level AI you tell it to solve the issue with the company’s philosophy in mind and wonder why off the shelf human level AI doesn’t have that in its “system prompt” or whatever term it will be.

This whole problem is blown out of something that every AI-phobe seems to assume but keeps secret, which is a common trope at this point.

[yetihehe]

> If it’s a human level AI you tell it to solve the issue with the company’s philosophy in mind and wonder why off the shelf human level AI doesn’t have that in its “system prompt” or whatever term it will be.

That IF is very big case here.

> This whole problem is blown out of something that every AI-phobe seems to assume but keeps secret, which is a common trope at this point.

It happens again and again CURRENTLY that AI's behave exactly like Djinns. They do what you tell them, not always what you mean. Like, "Geologists Recommend Eating At Least One Small Rock Per Day". They make perfectly coherent advice, but sometimes it's hilariously wrong and AI doesn't know it's wrong. But people trust it anyway.

[wruza]

Yes, but that IF was introduced by this subthread’s OP themselves. Statements like that make no sense if it’s possible and not applicable otherwise. And everyone seems to be ok with that on HN, regularly. Notice how inconvenient comments get simply ignored.

What current “AI” does is irrelevant in this context, but I think you’re overdramatizing on this tangent a little. Almost everyone knows that 2024 AI still hallucinates and you can’t take its output at face value. Those who don’t know may be still researching or just generally gullible/nonanalytic. There’s no problem here, taking our regular baseline into account. People believe yellow press and influencers everyday.

[matchagaucho]

It's turtles all the way down. C++ compiles to C which compiles to Assembly...

AI will own the next layer... dynamically generating SQL and code to access existing systems.

[BiteCode_dev]

Or maybe just delegate us the work. Maybe we are the assembly of AI and we don't know it yet.

[mewpmewp2]

But if the AI is human level then why not just let AI fix itself or create whatever fallback software needed itself?

[_wire_]

> Let's assume that AI is human level (it's not, but humour me)

OK

> Now something goes wrong. A customer isn't getting what they ordered. What do you do about it?

What do you do when something goes wrong with a human?

Do that.

End of story.

[danpalmer]

This is my point, you can't. You can't hold an AI accountable in the way you can with a human, whether that's accountable to a contract or accountable to the law, and the only ways in which you can hold an AI accountable involve escalating to a human, at which point we're back to where we are now without AI and requiring essentially the same software and business processes.

[paulddraper]

If it's human level, you can.

You train it, you replace it with a more suitable worker, whatever.

You've answered your own dilemma with the premise.

[devnull3]

Also AI (as promised) will deliver a customized solution to an extent that for a human to even solve the problems in it will need a lot of context. This in itself makes it dependent on humans. Also you have to factor in knowledge redundancy, non-availability of your personnel. So you will always have to account for more people than AI promises.

[ornornor]

You can reason with a human. You can’t with an AI.

And if you got rid of all your human tools because AI, how do you put a human back in the loop?

[portaouflop]

We assumed it was human level in this thought experiment. I don’t see why you wouldn’t be able to reason with a human level AGI

[ornornor]

Are there any consequences for an AGI?