Hacker News new | ask | show | jobs
by freedomben 849 days ago
It's also important to remember that these things can change in near real time. Someone running a query seconds later than someone else could be using a different model/code. Coupled with the general indeterminisitic nature of LLMs, it really means that not getting similar results isn't nearly the disproving that software engineers are used to. I hate it because trusting others or accepting non-reproducible things as evidence is deeply antithetical to my scientific approach to things, but it is what it is.
3 comments
digging 849 days ago
It also means that anyone can report any result at all and claim it's real, unfortunately. Without a reproducible receipt of conversation, any report of an AI conversation could be very easily faked.
link
jazzyjackson 849 days ago
it's really tragic that LLMs became normal consumer tech before public key signatures. It would be trivial for openai to sign every API response, but the interfaces we use to share text don't bother implementing verification (besides github, the superior social network)

i blame zoom for acqui-sunsetting keybase.

link
miki123211 849 days ago
Open AI actually solved this, you can share a link to a Chat GPT conversation and then it's trivial to verify that it's authentic. You can't fake Chat GPT output in one of these without hacking Open AI first.
link
freedomben 849 days ago
Good point, although how long do those links stay live? Do they get deleted after 30 days or anything? Any idea if OpenAI has ever deleted one, especially for violating content policy or something?
link
jazzyjackson 848 days ago
true enough, I guess my comment is more lamenting the culture around sharing screenshots instead of a verifiable source.

Similar story with DALLE3 / sdxl output - its the jpeg that gets shared, no metadata that might link back to proof of where it was generated (assuming the person creating the image doesn't choose to misrepresent it, if they want to lie and say it wasn't DALLE3 then we have to talk about openai facilitating reverse image search...)

link
solardev 849 days ago
LLMs work surprisingly similar to our politicians: Money in, customized bullshit out.
link