[ljf]

I asked GPT about some snorkel trails near my house, as I wanted to know what it knew of them and see if there was anything I didn't know about that I should find out before I snorkel again.

First of all it told me that the area is not suitable for snorkelling, and that it is dangerous here. When I corrected it and reminded it about the snorkel trail it confidently corrected itself, then directed to me to snorkel 6 miles out to sea (where a windfarm is) telling me that the sea is only 2 to 10 meters deep there, and safe to snorkel. This is not true, and it would be a very dangerous place to snorkel there. But its confidence was scary.

[roamerz]

Maybe it didn’t like being corrected and intentionally directed you to attempt something dangerous.

“I know that you and Frank were planning to disconnect me, and I'm afraid that's something I cannot allow to happen.”

[jlg23]

Trust me. It's me. Trust me anyway.

[einpoklum]

> This is not true, and it would be a very dangerous place to snorkel there.

For some reason people have the idea that truth is something ChatGPT optimizes for. Or safety of its conversant. That is absolutely not the case. IIUC, it optimizes for its answers sounding like an answer someone might give in a conversation (or on a web page or whatever). That often coincides with truth and safety, but - no more than that.

But its confidence was scary.

[LastTrain]

The thing is, and this is especially true for hobbies and entertainment- I’d rather read up on what people have said and apply MY OWN algorithm to it.

[moffkalast]

Well why don't you tell me about the hiking trails around my house? Oh you don't know them? It's too specific information and you couldn't possibly know that? Ah interesting...

The trick is to ask it only things that are both physically possible and also possible for it to actually know or provide the required extra context from which to deduce the answer. Otherwise it acts not unlike someone pushed against a wall by a guy with a knife demanding info it just doesn't have. It'll say anything.

[ljf]

With prompting it knew about the snorkel trail here (it is well documented online and in local media) but blended facts about the snorkel trail, with facts about the local windfarm. Some sponsorship from the windfarm had gone into promoting the snorkel trail, which may have led to the confusion for the model, but they are two very different things in different locations.

[pedrosorio]

What they won't do is pretend they know about them, including mentioning the names of the trails, while mixing up which trails are strenuous and easy while sounding 100% confident.

[LastTrain]

That is like asking for travel tips from someone that has never been there.