I can't be the only one who looks at this and doesn't think its that silly that it does that. I mean it's trying to incorporate a fact its being provided. Its insane it can do that at all. You could tune it to prefer pre-existing knowledge and not let the user correct it so easily, and to be more skeptical, but that would have downsides too. I don't think it's some big coup that you can tell it Google is a mushroom and it synthesizes that.
One of the worlds leading tech companies deployed a new search function "that our users really love!!!"; but when asked, told me that there are two letters 'n' in the word 'Google'.
Yes, it's because they're using a cheap model to answer my question. Yes, I know how a tokenizer works and why this happens. No, I don't think the tech industry is in an insane place at all, why do you ask? /s
A simple post, shows that a 5 trillion dollar scientific research project, that sustains the current market valuation that separates the USA from bankruptcy, can be defeated with a simple prompt manipulation.
Anyone can drive an F-22 into a ditch. Doesn't mean that it can't also be used to drop a 2k lb. bomb down your chimney from 40,000 ft.
That demonstration is interesting, but not really something new. Fooling very intelligent people into believing something completely absurd is incredibly easy. How many scientific papers have been retracted based on wholesale fabrications that fooled an entire review committee?
The question isn't "What is the dumbest thing I can do with this technology?" its "What is the most valuable thing I can do with this technology?"
The technology is so dumb can be easily made to believe there is a Google mushroom. We are way far from driving a F22 to the ditch...although I am sure with the same techniques, we could make the AI make the F22 bomb the Google headquarters....
A table saw is technology so dumb it can be made to chop off your fingers.
An air conditioner is technology so dumb that it can be used to kill an infant with hypothermia.
A human is a sentient being so dumb that it can be made to believe in things far more outlandish than a “Google mushroom”.
I can keep going. The point is that just about anything useful can do something dangerous or stupid. Most people can see that. Most people are more interested in how useful something can be, not how useless it is when intentionally misused.
Why is the comparison to someone being able to intentionally crash their own F22, to the above example of intentionally trying to get bad results from their cheapest AI, a bad one?