If you want constructive feedback and be told you are wrong then you need to craft a system prompt for it. They have been trained to be agreeable and tell you your ideas are great.
If you ask for an assessment of something, it's often as easy as adding "be critical" to your prompt. e.g.:
"What do you think of this short essay?"
"What do you think of this short essay? Be critical."
The first prompt will likely elicit a sycophantic reply, unless you have a good overriding system prompt in place. You'll always get much better feedback with the second.
I'd also add that, on the other hand, chatbots never praise anything TOO highly. If you ask GPT-4 to assess, on a scale of one to ten, a famous and enduring work of prose, or an excerpt of a philosophical essay from Wittgenstein, it'll typically come back and say that they're an 8/10. Rarely 9/10. Never 10/10, no matter what you submit.
Getting LLMs to argue or to ask for more context rather than always offering an answer are among the most difficult interactions to elicit. There's a big gap between helpful and obsequious, and unfortunately society often selects for teh latter.
"What do you think of this short essay?"
"What do you think of this short essay? Be critical."
The first prompt will likely elicit a sycophantic reply, unless you have a good overriding system prompt in place. You'll always get much better feedback with the second.
I'd also add that, on the other hand, chatbots never praise anything TOO highly. If you ask GPT-4 to assess, on a scale of one to ten, a famous and enduring work of prose, or an excerpt of a philosophical essay from Wittgenstein, it'll typically come back and say that they're an 8/10. Rarely 9/10. Never 10/10, no matter what you submit.