[simonw]

I tried to hint at that with "Using them as an alternative to a search engine such as Google is one of the most obvious applications—and for a lot of queries this works just fine."

I agree: I do use it as a search engine myself for a bunch of things, but those tend to be things where I've developed a strong intuition that it's likely to give me a reasonable result.

People who haven't developed that intuition yet tend to run into problems - and will often then loudly proclaim that LLMs are evidently useless and shouldn't be trusted for anything.

[lupire]

How useful is it if you have to know what you expect the answer to be? How do you know you are right in your calls of when to trust it? This smells like a confirmation bias machine.

[simonw]

It turns out to be absurdly useful, to a very un-intuitive degree.

One trick I use is to assume it has a "Wikipedia-level" knowledge of pretty much any topic. Often that's what I need! I want to ask some quick questions of someone eloquent who's read the Wikipedia article about something, to save me from having to read the whole thing myself.

If I need more expertise than you can get from reading Wikipedia I know that ChatGPT alone is very unlikely to cut it.