[mattlutze]

> exotic topics [...] I don't know how much

We also don't know, in situations like this, whether all of or how much of the research is true. As has been regularly and publicly demonstrated [0][1][2], the most capable of these systems still make very fundamental mistakes, misaligned to their goals.

The LLMs really, really want to be our friend, and production models do exhibit tendencies to intentionally mislead when it's advantageous [3], even if it's against their alignment goals.

0: https://www.afr.com/companies/professional-services/oversigh... 1: https://www.nbcnews.com/world/australia/australian-lawyer-so... 2: https://calmatters.org/economy/technology/2025/09/chatgpt-la... 3: https://arxiv.org/pdf/2509.18058?

[dangus]

Despite those mistakes, the utility is undeniable.

I converted some tooling from bash scripts leveraging the AWS CLI to a Go program leveraging the AWS SDK, improving performance, utility, and reliability.

I did this in less than two days and I don’t even know how to write Go.

Yes, it made some mistakes, but I was able to correct them easily. Yes, I needed to have general programming knowledge to correct those mistakes.

But overall, this project would not exist without AI. I wouldn’t have had the spare time learn all I needed to learn (mostly boilerplate) and to implement what I wanted to do.

[autoexec]

> The LLMs really, really want to be our friend

They want you to think they are your friend but they actually want to be your master and steal your personal data. It's what the companies who want to be masters over you and the AI have programed them to do. LLMs want to gain your confidence, and then your dependence, and then they can control you.

[dangus]

This seems hyperbolic to me. Sometimes companies just want to make money.

Similarly, a SaaS company that would very much prefer you renew your subscription isn’t trying to make you into an Orwellian slave. They’re trying to make a product that makes me want to pay for it.

100% of paid AI tools include the option to not train on your data, and most free ones do as well. Also, AI doesn’t magically invalidate GDPR.

[ToucanLoucan]

> This seems hyperbolic to me. Sometimes companies just want to make money.

It's not hyperbolic at all. The entire moat is brand lock-in. OpenAI owns the public impression of what AI is- for now- with a strong second place going to Claude for coders in specific. But that doesn't change that ChatGPT can generate code too, and Claude can also write poems. If you can't lock users into good experiences with your LLM product, you have no future in the market, so data retention and flattery are the names of the game.

All the transformer-based LLMs out there can all do what all the other ones can do. Some are gated off about it, but it's simulated at best. Sometimes even circumvent-able with raw input. Twitter bots regularly get tricked into answering silly prompts by people simply requesting they forget current instructions.

And, between DeepSeek's incredibly resource-light implementations of solid if limited models, which do largely the same sort of work without massive datacenters full of GPUs, plus Apple Intelligence rolling out experiences that largely run on ML-specific hardware in their local devices which immediately, full stop, wins the privacy argument, OpenAI and co are either getting nervous, or they're in denial. The capex for this stuff, the valuations, and the actual user experiences are simply not cohering.

If this was indeed the revolution the valley said it was, and the people were lining up to pay prices that reflected the cost of running this tech, then there wouldn't be a debate at all. But that's simply not true: most LLM products are heavily subsidized, a lot of the big players in the space are downsizing what they had planned to build out to power this "future," and a whole lot of people cite their experiences as "fine." That's not a revolution.

[autoexec]

> Sometimes companies just want to make money.

Companies never just want money, because more power means more money. Regulatory capture means more money. More control means more money. Polluting the environment and wasting natural resources means more money. Exploiting workers means more money. Their endless lust for money causes them want all sorts of harmful things. If companies were making billions and nothing was being actively harmed by any of it no one would care.

These companies do want your money, but once you're locked in you are no longer the customer. If these AI companies had to depend on the income they get from subscriptions to survive they'd have gone out of business years ago. Instead AI is just shoved down people's throats everywhere they look and the money these companies live off of is coming from investors who are either praying that the AI becomes something it isn't or they're hoping they can help drive up stock value and cash out before the bubble breaks and leave somebody else holding the bag.

0% of AI tools include the option to not train on my data. They've already stolen it. They've scraped every word and line of code I've ever written that's been transmitted over the internet. It's been trained on photos of my family. It's been trained on the shitty artwork I've sent to my friends. By now it's probably been trained on my medical information and my tax records.

AI is controlled by some of the most untrustworthy companies and people on earth who have been caught over and over lying to the public and breaking the law. They can promise all day long not to steal anything I voluntarily give them, but I have zero trust in them and there is no outside oversight to ensure that they will do what they say.

The people behind what passes for AI don't give a shit about you beyond whatever they can take from you. They are absolutely not your friend. AI is incapable of being your friend. It's just a tool for the people who control it.

[dangus]

I feel like you’re still using hyperbole here. For example, you said your family photos were used for training, but most cloud photo providers specifically tell you in their privacy policies (legally binding) that they don’t do that.

My family photos have never trained AI, because my iCloud Photos service specifically says they don’t do that and explains the technical implementation of their object recognition system in detail. Apple even offers an e2e encrypted mode of operation. (Still, I have now moved to a more customer-friendly solution away from iCloud).

As far as training on your code, well, you either believe in open source or you don’t. AI training doesn’t even violate the most copyleft open source licenses. Unless AI has reproduced your code verbatim it’s not engaging in any kind of copyright reproduction.

[jjav]

> 0% of AI tools include the option to not train on my data.

That's perhaps not true. If you sign up for the enterprise accounts there are options to not use any of your data to train. That's how we have it set up at $job.

(I say "perhaps" because of course I'm still sending all the data to the AI and while the contract has an ironclad clause that they won't use it, there's no way to 100% verify that.)

[tacet]

They mean the data AI companies scrape(d) to train their models.

For example they can't opt for their comment not to be scraped off HN and used for training.

[marcellus23]

I wonder if, in any of those legal cases, the users turned on web search or not. We just don't know -- but in my experience, a thinking LLM with web search on has never just hallucinated nonexistent information.

[andrepd]

I'm sorry to be so blunt but this is a massive cope and deeply annoying to see this every. fucking. time. the limitations of LLMs are brought up. There is every single time someone saying yeah you didn't use web search / deep thinking / gpt-5-plus-pro-turbo-420B.

It's absurd. You can trivially spend 2 minutes on chatgpt and it will hallucinate on some factually incorrect answer. Why why why always this cope.

[andybak]

The 2 minutes thing feels out by one or two orders of magnitude.

But then - I'm constantly amazed by how everyone's subjective and presumably honest accounts of their experiences with AI differ so wildly.

[andrepd]

You're right, it does. It's more like 10 seconds.

[1718627440]

Well I agree with you that LLMs really like to answer with stuff, that is not grounded in reality, but I also agree with the parent, that grounding it in something else absolutely helps. I let the LLM invent garbage how ever it feels like, but then tell it to only ever answer with a citing valid existing URLs. Suddenly it generates claims, that something doesn't exist or it truly doesn't know.

This really results in zero hallucination (but the content is also mostly not generated by a LLM).

[andrepd]

Well I don't know what to say, except that this is obviously, trivially, not true. The LLM will plain make up links that don't exist, or at least "summarise" an existing link by just making stuff up that is tangentially (but plausibly) related to the link. It's impossible to have used LLMs for this purpose for more than a quarter of an hour and not have seen this.

[1718627440]

I never had the case, that an URL did not exist. For me it shows stuff like "generating web search", so I guess it tries to fetch the URL first, before suggesting it. LLMs like to give tangentially related links, but this is typically paired with a sentence, that the link I really asked for, does not exist.

> It's impossible to have used LLMs for this purpose for more than a quarter of an hour and not have seen this.

You may be generalizing too much from your experience.

[marcellus23]

Maybe you're seeing this argument come up all the time (and maybe everyone else in this thread is disagreeing with you) because your experiences actually don't reflect everyone else's. I guess the other alternative is we're all morons and you're the only smart person here.

Also: if it's so trivially reproducible, then can you provide a ChatGPT transcript link of this happening?

> I'm sorry to be so blunt but this is a massive cope

Coping for what? I don't work for an AI company. If AI vanished tomorrow I wouldn't particularly care.

[noosphr]

>We also don't know, in situations like this, whether all of or how much of the research is true.

That's perfectly fine since we don't know how much of the original research is true either: https://en.wikipedia.org/wiki/Replication_crisis

If I waste three months doing a manual literature review on papers which are fraudulent with 100% accuracy have I gained anything compared to doing it with an AI in 20 minutes with 60% accuracy?

[Jensson]

> If I waste three months doing a manual literature review on papers which are fraudulent with 100% accuracy have I gained anything compared to doing it with an AI in 20 minutes with 60% accuracy?

You don't see how adding 40% error rate on top of that makes things worse? Your 20 minute study there made you less informed, not more, at least the fraudulent papers teaches you what the community thinks about the topic while the AI just misinforms you about the world in your example.

For example, while reading all those fraudulent papers you will probably discover that they don't add up and thus figure out that they are fraudulent. The AI study however will likely try to connect the data in those so they make sense (due to how LLM works, it has seen more examples that connect and make sense than not, so hallucinations will go in that direction) then the studies will not seem as fraudulent as they actually are and you might even miss the fraud entirely due to AI hallucinating arguments in favor of the studies.

[noosphr]

You are assuming my time is free and then comparing the difference between spending infinite time on something and minimal time.

[Jensson]

Uninformed is better than misinformed, its better to not do that research at all than having such a high error rate as your example had. AI models often have much less error rate than you said there for certain topics, but the 40% error rate in your example does firmly put it where you are better off doing nothing at all than using that for research.