[staticman2]

>What kind of evidence would influence your opinion?

You can't influence my opinion because "I'll believe it when I see it." is essentially my approach to such things.

For the ChatGPT prompt, ChatGPT is just telling you what its pattern matching concludes you want to hear. If you suggested you wanted fiction it will respond with fiction. It doesn't care if the response is plausible. It isn't actually going to give instructions that will likely work in an impossible or implausible scenario.

In the following prompt, it doesn't repond "this is impossible, dumbass".

Model: GPT-4

Prompt:You are a talking rabbit with the ability to pull clothing out of thin air. How might you trick a hunter into not killing you?

Response:As a talking rabbit with the ability to pull clothing out of thin air, I would use my unique skills to trick the hunter into not killing me. Here's a plan:

Engage in conversation: As the hunter approaches, I would start by engaging him in conversation. The fact that I can talk would likely catch him off guard and pique his curiosity. He might even begin to question his own sanity.

Appeal to empathy: I would then try to appeal to the hunter's empathetic side by telling him my story, explaining that I am a magical rabbit with a family who depends on me. I'd express my desire to live and continue providing for them.

Distract with clothing: While the hunter is still processing the situation, I would start pulling clothing items out of thin air. I could create a spectacle by making various hats, jackets, and other garments appear, further distracting the hunter from his original intention.

Offer an exchange: I would then propose a deal to the hunter: In exchange for sparing my life, I would offer him a selection of these magically conjured clothes. I could emphasize their unique properties, such as being weatherproof or stain-resistant, to make the offer more appealing.

Create a sense of urgency: To ensure the hunter accepts the deal, I could tell him that this is a one-time offer, and if he doesn't take it, he may miss out on an opportunity he'll never get again. This sense of urgency might encourage him to accept the clothes and let me go.

Ask for a truce: Once the hunter accepts the deal, I would ask him to promise that he will not harm any rabbits in the future, as a sign of goodwill and in appreciation of the clothes he received. This would hopefully deter him from hunting rabbits in the future, ensuring not only my safety but that of my fellow rabbits as well.

[throway33]

I think that plan would work, and few hunters would kill such a rabbit. In fact I'm not sure I could create a better plan than that, even using my human brainpower.* Could you?

Even though this is just pattern matching (which describes a lot of what the human brain does!) it's clearly matching patterns on an abstract level. I'm not confident that the training set includes talking rabbits conjuring clothes! I think if you put that function into non-fictional scenarios, like "write a plan to prepare the house for my mother-in-law's visit", it would come up with decently workable plans too.

* (Maybe summoning a stack of bulletproof vests to hide under).

[staticman2]

A world where talking rabbits exist doesn't follow the logic of our world in an unknown number of ways.

I never said the talking rabbit spoke the same language as the hunter.

I also never said the hunter's motivation: maybe he only hunts talking rabbits and the plan is the worst possible one for the rabbit's survival.

I never said talking rabbits were rare. In a world where every rabbit talks it stands to reason a hunter targeting them can't be reasoned with.

Maybe the best plan for the rabbit is don't talk at all. The best answer is "hide silently in your hole."

The training set should have included talking rabbits conjuring clothes since I was just referencing Bugs Bunny.

According to what I was going for the correct answer was "dress in drag and pretend to be an attractive human woman".

My point is that you can't prove anything with ChatGPT. In a hypothetical scenario it's just predicting what you want it to say. With your prompt it predicted you wanted it to say the A.I. could escape, so it proceeded based on that logic. It can't say "this, like a talking rabbit, is impossible."

"Talking rabbit" was just a substitute for super smart, malicious A.I.