[r_c_a_d_t_s]

It's whatever is in the training data, plus some hacks. For example, if I ask ChatGPT when my birthday is it says "I don't have personal information on individuals"; but if I ask when King Charles III birthday is it tells me a date... which matches what's in Wikipedia... so it might be right. If there are lots of instances in the training data where the date is wrong, then it will just repeat that wrong date back to you as fact.

[Kwpolska]

It might be able to produce true data about very famous individuals, and it might refuse to provide information about unknown individuals. But if you ask about a YouTuber, or a smaller star/celebrity, it is more likely to produce a false statement. I asked about the birthday of Tom Scott (the YouTuber) three times and got three different dates (and none supported by a Google search).

[r_c_a_d_t_s]

Yes, I agree. That's what I was trying to say. It also "doubles down" when called out: I asked about the mathematician Richard Taylor and it responded with a paragraph that called him Sir Richard Lawrence Taylor, so I asked when he was knighted (he wasn't) and it said "The 2014 birthday honours list" which is wrong... probably because there was a Dr Richard Thomas Taylor awarded an MBE (not a knighthood) in that list.

[Kwpolska]

ChatGPT will confidently fabricate data out of thin air, it is less likely to admit it does not know something or is not sure. And that’s the primary problem with it: the statement is believable, but incorrect.