[jcgrillo]

It should have flatly refused. If you gave a product like that to customers you'd be exposing yourself to unbounded downside liability risk. It's a completely nonviable technology for that kind of application, unless you can somehow make it have judgment. But you can't, because it doesn't reason.

A reasonable travel agent would have fired me as a customer. The LLM failed to do so.

[JohnBooty]

    It should have flatly refused.

I disagree in the strongest possible terms.

I think the LLM should advise you of risk and lack of feasability but should otherwise answer the question, unless you're trying to do something plainly destructive to others e.g. weaponizing anthrax or something.

    A reasonable travel agent would have fired me as a customer.

Unless the LLM was actually acting as a travel agent -- booking the trip for you -- as opposed to merely advising you, this expectation feels off.

    unless you can somehow make it have judgment

It did have judgement. It told you what a bad idea it was.

I think this is a great example of the unrealistic expectations people have for LLMs. No sane and sensible person would treat any single source of knowledge as infallible, for any consequential decision.

(Certainly, of course, you don't have to look very far for examples of idiots being overly trustful of LLMs, or Google, or GPS, or Wikipedia, or whatever. It certainly does happen and yes, I've heard all these arguments before about other technologies besides LLM. Replace "LLM" in your post with any of those other terms, and I promise you somebody made literally the exact same argument in 2003 or 2009 or 2014 or whatever)

Any reasonable person would consult a second doctor, or at least other sources of knowledge, after the doctor advises them of some irreversible course of action. Because we don't even expect highly trained and intelligent medical professionals to be perfect.

And yet, we get angry at LLMs for not having perfect judgement, even though their creators are extremely literal about how they can make mistakes.

[jcgrillo]

All I'm really saying is that if you want to try to automate a travel agency, LLMs ain't gonna get it done. They'll happily book you a really unsafe trip. So the technology doesn't work in this domain. The whole, empty promise is that this thing is supposed to automate jobs like travel agent away. But it can't. This isn't a "pro" or "anti" position, it's simply that there's no market for the technology here. Or anywhere else (like radiology) where actual responsibility and judgement is important. In fact, I can't think of a single job where it's optional.