Slight sidebar based on the content of the article. I don’t like the term “hallucination” for when a LLM produces nonsense. As if it otherwise has some grasp of reality and when it is wrong it is because it is hallucinating. Everything it produces is a “hallucination“, some of those are just more useful than others.
It was a marketing term that worked both ways. In my company they don’t trust AI because, well, it doesn’t actually work it’s just really good at being lucky. Which is fine for a lot of things, and absolutely horrible for things which aren’t fault tolerant. After LLMs got things wrong on a few contracts the top issues a company wide AI ban (outside of IT anyway, but that’s mainly because they hardly know what IT does).
I’m not sure we really suffer from it. In our internal analytics AI tends to slow employees down and make them less productive. This is in general. The exception is experts using it, where LLMs increase their value output by an ok margin. Especially within our own programming team LLMs have proven a real challenge. On one hand they are fancy auto-complete which will speed an experienced developer up by so much it’s hard to ignore. On the other hand it takes an experienced developer to know when they get things wrong. Not the things that literally won’t work, but the things that will work poorly. I haven’t been too involved outside of our team, but I imagine it’s the same in any field.
Which is where the “hallucination” term sort of back-fired. It was a good way to make people buy into the value of LLMs by making the mistakes oddities almost negligible. The issue is that those mistakes can have such massive impacts that the entire trust in the AI industry falters. I mean, we had one of the CEOs ask if we could switch to Linux now that Windows includes AI… obviously we can’t do that without going bankrupt, but it tells you something about the worry in the non-tech enterprise top.
Engineering is something you calculate and get expected results with some acuracy. E.g building a bridge without trials and errors (hopefully). I would prefer prompt “tailoring” which is more like you start with something general and try to fit it to your desired output with many many trials and errors.
It's more like "social engineering" and other uses of the word which don't directly deal with engines, which is what "engineer" used to mean. For example, Google lists one of the definitions as "the action of working artfully to bring something about".
I think it's fair to say that these models may have some grasp of reality insofar as the data we collect ballparks reality, and also insofar as the mechanism to learn from the data effectively extracts the truth value of the data.
We might say the same thing about people.
Ultimately, just how problematic is it to label something as a hallucination? Are investors about to be massively duped? If I create a mechanism to reduce hallucinations and I call it therapy, is that really problematic?
> I think it's fair to say that these models may have some grasp of reality insofar as the data we collect ballparks reality, and also insofar as the mechanism to learn from the data effectively extracts the truth value of the data.
No, it would be fair to say they have a "grasp" of predicting the next word in a given sequence of words based on a set of words in their training set. Hallucination then is what people call their inherent tendency to run into a situation where the probability of the next word being predicted "wrong" is high. And once one "wrong" word has been predicted the probability that the next word is also "wrong" rises exponentially.
LLMs do not have any grasp of reality. They just predict text based on trained patterns. Too many people have been fooled into believing that LLMs can understand anything about reality, but a word-based description of reality is not the same as reality.
> We might say the same thing about people.
If you want to reduce a human being down to being a word predictor, then I guess you could say that?
You don't understand anything about reality. For all you know, you're living in Plato's cave. What you kind of know, is just text you read from a physics book. The things your eyes see, could just as easily be encoded as tokens and fed into an LLM.
Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback - https://arxiv.org/abs/2305.14975
You're probably not wrong. But I think we should recognize that what is humanity is rather fluid. If you were to talk to the 15th century clergymen, I think they would tell the horrors of printing press and how it has/will break humanity. And they wouldn't be wrong either. Humanity evolves, and, well, that's it. And overall, hopefully, and this is where we hopefully can effect positive change, we can steer the change to more humane, and moral direction.
I think they simply shone a light on human nature. Those aspects were there and still affected people, but we had the illusion that we were rational humanist actors - and even then that was only in the west. the internet simply shattered the illusion.
This isn’t a good analogy. The internet provably works. AI doesn’t provably work. It works some of the time, and there’s no indication we are anywhere close to a version of AI which provably works all of the time, or even up to human standard.
I was just fixing some php from 2001 and it allows visitors to execute php, inject js, read/write the database, send emails, extract md5 passwords, extract email addresses.
Probably just companies trying to impede progress of other companies. Not to say that the statement is wrong necessarily. But given that this is coming from a group of people that could very easily solve the problem, I'll take it with a grain of salt.
It’s not an easily solvable problem. They can’t make an AI which won’t lie or make stuff up, which is sort of the root of the problem. Imagine an AI which is granted access to control systems. We can’t trust such an AI to run control systems any more than we can trust it not to lie or make stuff up. There isn’t the sort of rigor behind AI development to permit creating a provably correct AI. There needs to be more study in order to understand the limits of AI fallibility and failure modes.
They could just collectively stop working on the problem until they feel that the issue is resolved (moratorium). That's what I meant by "they could very easily solve the problem".
Or they could still work on it, but don’t use it on customers until it’s good enough.
> Zhou shared a striking example of how AI-generated content could lead to real-world consequences. “Some of the initial stock images of various ingredients looked like a hot dog, but it wasn’t quite a hot dog—it looked like, kind of like an alien hot dog,” he said. Such errors, he argued, could erode consumer trust or, in more extreme cases, pose actual harm. “If the recipe potentially was a hallucinated recipe, you don’t want to have someone make something that may actually harm them.”
There’s absolutely no reason Instacart has to show customers AI-hallucinated recipes from stock images. They choose to do it, then beat the drum about AI security as if they actually give a shit. It’s like Boeing self-certification.
There are some techniques to alleviate hallucination, contradictory or confusing answers, but I have difficulty imagining a provable correct LLM because the attack surface is so large.
The current methods to train for AI safety might be augmented with insights from chaos engineering, cognitive psychology, marketing and persuasion - making them agogic truth machines scoring very low on hallucination benchmarks [1].
I think we should program and train LLM with universal recognized agogic principles instead of being neutral in this regard, to encourage critical thinking and prevent 'reality tunnels' in the mindset of the users and perhaps incorporating this also in future training and curating techniques [2][3][4]. How to raise GenAI and future AGI well.
There are LLM training techniques to alleviate hallucinogenic, contradictory and confusing answers. These might be augmented with insights from chaos engineering, cognitive psychology and persuasion - making them agogic truth machines scoring very low on hallucination benchmarks [1].
I think we should program and train LLM with universal recognized agogic principles instead of being neutral in order encourage critical thinking and prevent 'reality tunnels'. Perhaps incorporating this in future training and curating techniques [2][3][4]
* Data curation
Ensure data used to train AI models is balanced and diverse helps in preventing biases that could lead to hallucinations or harmful outputs. So curating data from a wide range of sources, cultures and viewpoints.
Implementing quality control during data collection and preprocessing to filter out unreliable, outdated, or biased information.
* Targeted post-training (fine-tuning)
After initial training models can be fine-tuned using datasets specifically designed to emphasize helpfulness, harmlessness and alignment with ethical principles. Embed ethical guidelines in datasets, for example include scenarios to handle sensitive topics, avoid hate speech and promote fairness.
* Red-teaming
Red-teaming involves stress-testing the model by simulating adversarial attacks or intentionally providing challenging prompts to see how the model responds. This helps identify weaknesses, such as susceptibility to generating harmful content or hallucinations. This can be used to improve the model's robustness and safety.
* Post-training datasets focused on responsible AI principles
Incorporating datasets that help the model understand context and nuance of various topics, ensuring it can provide appropriate responses to the situation.
* Refusal-aware instruction tuning
While data curation, targeted post-training, and red-teaming help to prevent the introduction and propagation of false or harmful content, R-tuning directly enhances the model's ability to recognize its limitations. Enabling the model to refuse to answer questions beyond its knowledge.
* Iterative user feedback based refinement
Continuously collecting and analyzing feedback from users and independent review teams helps identify issues that may not have been apparent during development.
As commented before, by say that they "could very easily solve the problem" I meant that they could just collectively stop using the problematic AIs in prod until they feel that the issue is resolved (moratorium). Not that it's easy to resolve the technical difficulties.
Make companies and their leadership responsible for any harm or damage that comes from AI generated content or actions. If a recipe is a hallucinated recipe and poisons someone, treat it the same as if someone created a salad recipe calling for raw kidney beans and rhubarb leaves in order to intentionally harm someone.
AI creates a ton of garbage unmitigated, and cracking down on when that garbage is harmful will be a good way to reduce the amount of garbage these companies put out.
If a company has an AI tool, then you must treat everything that comes out of that tool as if it came from the company directly. If something harmful comes out, then the product is defective and they must compensate users like any other defective product.
If a hammer manufacturer sometimes produced a hammer that exploded like a hand grenade on impact, they would be held liable for that.
Someone on Twitter said: "Do not fear AI. Fear the people and companies that run AI".
After all, it's the same industry that came up with pervasive and invasive tracking, automated insurance refusals, automated credit lookups and checks, racial ...ahem... neighbourhood profiling for benefits etc. etc.
No need to fear anything. Will be handled similar to Parasites and Pandemics. Scale is over rated, if the host gets drained and dies too quickly. Past couple decades have taught corporate wonderland how to combine mindlessly ambitious people(parasites) and button press scalability, to generate mesmerizing levels of profits. But the scheme is getting old. Everyone has copied the model. The host is taking more hits than it can afford. And therefore there is greater recognition about which parasites to keep locked up and which can be safely controlled.
Puff piece wherein it’s revealed that the fix for spending lots of money to put ai in charge of things it obviously can’t do properly is spending a lot more money on “ai security” that just points out that ai isn’t working, and seems to have no real path towards fixing the problem.
The security aspect mentioned here is that ai generated recipes could poison you when they hallucinate. The fix is “governance” which isn’t really described or defined, but no doubt it’s as necessary as it is costly. We could probably just not use cooks that seem to randomly poison the food and not create a new industry of equally suspect chef-policing but hey, where’s the fun in that?
> “If you plan for the models and the chatbots that exist today… you’re going to be so far behind,” he reiterated, urging companies to prepare for the future of AI governance.
This is the key. Folks are looking at current capabilities rather than the trend line. We need to be ahead of the development AI. There probably ought to be laws regarding how AIs can be used or not used. There probably ought to be required disclosure when AI is used to create a work of art.
> There probably ought to be laws regarding how AIs can be used or not used. There probably ought to be required disclosure when AI is used to create a work of art.
There's no point fearing or obsessing over the inevitable. Pretend it already happened and adapt. A lot of the sentiment here is aimed at putting the cat back in the bag and then hoping for the best. It won't work. Not even a little bit.
Laws are going to do absolutely nothing here except slow down the wrong people. The world is full of people who don't play by rules. It's a level playing field already. Everybody is doing AI at this point: the Iranians, the Chinese, the North Koreans, criminals, sociopath entrepreneurs in Silicon Valley, and everybody else with bad intentions you can think about. Every idiot with half a clue as well.
Laws are for obedient citizens that aren't a threat. It's all the other people we need to worry about. Besides, US law is worthless across its borders. And I don't even live there. I live in a place with actual anti AI laws (Europe) and I think they are a really bad idea. I'd prefer to be in a place that isn't awaiting the inevitable meekly with their hands tied behind their backs.
There is a nice little howto build and LLM from scratch article featuring on the HN front page today. Assume everybody else read that too. And then some. This is all public knowledge at this point. You can get pretty far coding an LLM with the help of an LLM. Even the offline ones are probably not that bad for this. Any idiot can build an LLM at this point. Even me probably. So that means that world + dog has been figuring out how to use LLMs to their advantage for the last decade or so. Probably with a sharp uptick in interest about 2-3 years ago. We've seen nothing yet.
Given all that, the safest course of action is working on the assumption that this already happened or if not that it will be happening very soon. The trick is not to prevent the technology but to be better at it than everybody else. This is an arms race and we can't afford to let the wrong people win it.
Regarding the fears about art and impersonation. Same thing. That too is going to happen. Nothing we can do about it. No amount of virtue signalling is going to help here. So, I would propose the exact opposite. Let people do what they will (again, inevitable that they will). But require them to sign all their work and assume any unsigned work to be fraudulent, malevolent, and worthless. Illegal even (laws can work to our advantage here). AI work should be signed too. So we can trace it back to who, when, and how. AI will be able to fake a lot of things. But not cryptographic signatures.
It's a lot easier to regulate proper use of signatures than abuse of technology. And the beauty of this is that we've had the technology to do this for decades already. But we still send unsigned emails, publish unsigned articles on blogs, and don't bother with encryption in general, etc. This is stupid and it has been for quite some time. It's only the unsigned work that's easy to fake.
That's what sites like GitHub and Twitter already do. When I push to GitHub it's signed by me. When I tweet about something I did, that's a signature too. Would you like to be bold courageous one who writes a browser extension to block everyone and everything that's anonymous on these services? That's basically your vision. Google tried to make that happen with Google Plus but the service was sadly shut down.
I've been thinking about such a thing. It wouldn't be hard technically but it seems people are a bit indifferent on this front which makes it hard to pull off from a business point of view. We need some good scandals to get people a bit more paranoid.
Some scandals around LLM generated content could be exactly what we need here. I'm talking reputation damaging scandals that have the likes of the NYT having to explain in public how they messed up so badly and paying millions in damages to the victims. Money is a great way to incentivize companies to level up their game. The the likes of the NYT need to get paranoid about checking authenticity. And everybody else as well.
A few geeks wearing tin foil hats aren't much of a solution. PGP flopped for that reason. But it's not to late for it to make a comeback in some form.
IMHO the Fediverse is an obvious place to start. Why accept any content there that isn't signed? It should be stupidly easy to level up its protocols to add and verify signatures to content and profiles. I've actually considered having a go at it at some point. No time and other priorities. So, I havent. But why isn't this a big topic in the wider community?
I doubt there's going to be any such scandals, unless someone creates them. What's going to happen is LLM generated content is going to become so good so fast that you won't want to consume content created by people anymore, and you're going to feel guilty about that. Similar to how people probably feel guilty about playing football video games rather than going outside and playing football. Crypto and authentication services won't help, because people will just give their LLM agents the ability to act autonomously under their identity.
I can't believe anybody honestly thinks that building a powerful AI is honestly a good thing. It seems that we're all trapped in a "keeping up with the joneses" style race where even if every individual person agrees that building AI is a bad thing, they're not able to stop because they still want to beat the competition and reap the rewards. And once there are millions or billions of these AI agents running around each of whom is smarter than every human on earth, good luck trying to predict or control them..
Some people say they do, you know, summon the thermodynamic god and /wipe out all of humanity/ optimize the light cone according to the Jarzynski-Crooks fluctuation theorem.