There was a lot of stories like "Webb captures it's fist ever picture of an exoplanet" [eg]. My guess is that it's digesting those and not understanding that the "it's" in that sentence is critical.
Did you mean "its" such as in <https://news.ycombinator.com/item?id=34359839>? Given your statement of this being critical... :) (Advice I also gave at work today: just don't use contractions and the right spelling will usually be obvious. In an informal setting, it's more tempting, but that's the way to easily check yourself.)
I like that this thread points out even humans have difficulty with that construction sometimes. We're trying to hold Google's language model to a higher standard than humans in this case I think. I remember "learning" thousands of bits of trivia like that from people who had misinterpreted something they read and misstated it in such a way.
Of course Google has already been putting often-incorrect summaries/factoids in its search infoboxes for a few years now.
It's not a matter of "its vs it's" in this case, but the very existence of the word in the sentence:
"NASA’s Webb Takes Its First-Ever Direct Image of Distant World"
It doesn't matter if one misspells its. You know what it means and it largely defines this sentence. A failure to parse such a relatively simple construct doesn't bode well.
You're of course correct, but perhaps I should have focused more on the concept of skipping words when reading, misremembering, and "reading what you want to read" - those traits are extremely common if not universal at some level in human readers as well.
Well yeah, when I make a tool I want it to do its job correctly. If it doesn't I throw it out. If a human keeps messing up I do the same. A human messing up confidently in their interview probably won't even get hired...
Sure but this seems analogous to creating a claw hammer then showing off how it can be used to drive screws, then saying the hammer isn't doing its job correctly when the screws aren't driven properly.
I think chat technology like this is an incredible tool, but I don't think it's being judged fairly: I don't think its usefulness is as some kind of oracle or advisor expected to provide correct or logical information. That seems so orthogonal (if not diametrically opposed) from its actual function that it really feels like we're being trolled by things like "Galactica". But I'm much more (cautiously) optimistic about the potential use for the technology in web search, which has never been logical or "correct" and has always required critical thinking on the part of its users.
Perhaps there should be more of a disclaimer that the things it says are not and cannot be construed to be factual, no matter how verisimilitudinous.
My thoughts exactly. natural language semantics is imperfect and human reasoning is weird. Let's not mistake LLM models as a single source of absolute truth, but a funny & bullshitting assistant who happens to read and vaguely remembers much information.
I can't tell from the tweet why the Bard response is wrong. Is it because some other instrument has taken an image of an exoplanet, or because no instrument has ever done so? ChatGPT seems to believe it is the latter.
For what it's worth, thats not accurate - a spectrograph was used in 1995 to discover the first exoplanet by Mayor and Queloz, and they received a Nobel prize for this. So GPT is right. One can be pedantic and say that not an "image" but most would disagree.
i'm not a grammar expert but i think as a layperson there is a little bit of ambiguity in how a reader could interpret Bard's response. If i were writing i would probably specify "first ever picture". I think a lot of times words like "the first" are relative and context specific which the Bard answer lacks.
There is no ambiguity. We don't call Neil Armstrong "the first ever man on the moon". "The first" has a clear meaning and making any qualification of that implicit is either deceptive or bad writing.
i guess my take was that i find this a more excusable grammatical mistake than say getting a simple addition question wrong, which we've seen from gpt3. i could agree with "deceptive or bad writing" and maybe that's a more sinister error than something obvious wrong now that i think about it.
it's funny for me to think that i'm going out of my way to try and give Bard the same benefit of the doubt and work to try and find a way it could be right, the same way i would for a friend.
