[sanex]

We have an enterprise cursor account so I can try all the mainstream models. Using composer 2 on our own code which I obviously have the source code for I couldn't get it to turn on a debug flag to bypass license checks while I was troubleshooting something. Infuriating. It was like that old Patrick from SpongeBob meme.

I don't understand why we would turn the models into law enforcement officers. Things that are illegal are still illegal and we have professionals to deal with crimes. I don't need Google to be the arbiter of truth and justice. It's already bad enough trying to get accountability from law enforcement and they work for us.

[oneseven]

They're probably worried about liability. Let's say that Oracle finds out you reverse engineered their DB using Gemini. You can be sure they will sue Google. Not just for providing the tools, but you could make the argument that it's actually Gemini doing the reverse engineering, and on Google's hardware no less.

[Wowfunhappy]

Let's say that Oracle finds out you reverse engineered their DB using IDA Pro. Would you expect Oracle to sue Hex Rays?

I don't understand why everything changes as soon as an LLM is involved. An LLM is just software.

[sunnybeetroot]

The difference is IDA Pro doesn’t do something unless you instruct it to, an LLM is unpredictable and may end up performing an action you did not intend. I see it often, it presents me options and does wait for my response, just starts doing what it thinks I want.

[ethbr1]

This. It's going to be tricky for the frontier model labs to argue they didn't intentionally design their models to do so, when the models take illegal actions.

I'm not even sure how one would construct a viable legal argument around that for SOTA models + harnesses, given the amount of creative choices that go into building them.

It'd be something like "Yes, we spent billions of dollars and thousands of person-hours creating these things, but none of that creative effort was responsible for or influenced this particular illegal choice the model made."

And they're caught between a rock and a hard place, because if they cripple initiative, they kill their agentic utility.

Ultimately, this will take a DMCA Section 512-like safe harbor law to definitively clear up: making it clear that outcomes from LLMs are the responsibility of their prompting users, even if the LLM produces unintended actions.

[Wowfunhappy]

> I'm not even sure how one would construct a viable legal argument around that for SOTA models + harnesses, given the amount of creative choices that go into building them.

I'm not a lawyer, but to me the legal case seems pretty obvious. "We spent billions of dollars creating this thing to be a good programmer, but we did not intend for it to reverse engineer Oracle's database. No creative effort was spent making it good at reverse engineering Oracle's database. The model reverse-engineered Oracle's database because the user directed it to do so."

If merely fine-tuning an LLM to be good at reverse engineering is enough to be found liable when a user does something illegal, what does that mean for torrent clients?

[ethbr1]

> No creative effort was spent making it good at reverse engineering Oracle's database.

That's the bit that's going to be nasty in evidence. 'So you didn't have any reverse engineering in your training or testing sets?'

[jodrellblank]

> “making it clear that outcomes from LLMs are the responsibility of their prompting users, even if the LLM produces unintended actions”

So if I ask “how does a real world production quality database implement indexes?” And it says “I disassembled Oracle and it does XYZ” then I am liable and owe Oracle a zillion dollars?

Whereas if I caveat “you may look at the PostgreSQL or SQLite or other free database engine source code, or industry studies, academic papers; you may not disassemble anything or touch any commercial software” - if it does, I’m still liable?

Who would dare use an LLM for anything in those circumstances?

[nullstyle]

If they thought they would succeed, no doubt oracle would sue. I expect bad behavior from multinationals, especially oracle

[lokar]

They would not even expect it to succeed, just make an example of the company (the lawsuit is the punishment) to discourage others.

[sanex]

We need that lawsuit to happen already so we can establish precedent. The person in the driver's seat of the Tesla should be at fault. The engineer using the llm should be at fault. The person behind the gun not the manufacturer should be at fault.

[Iolaum]

We shouldn't need a lawsuit. The legislative branch should pass a law clarifying those things, that's their job.

[jon_richards]

Then you need a lawsuit to determine whether the law is “constitutional”.

[hvb2]

> The person in the driver's seat of the Tesla should be at fault.

I don't think this is a good analogy. For Tesla right now it might fly. However, when their software gets to waymo level of autonomy, I would expect liability to shift to the manufacturer.

If anything, I think that would be the true proof of a company trusting their software to allow for autonomous driving

[rokob]

> However, when their software gets to waymo level of autonomy

Luckily that won’t happen.

[kelvinjps10]

Also especially if they claim they're selling autonomous cars

[dotancohen]

I believe that Mercedes does offer manufacturer liability.

[missedthecue]

In the America, whoever has the most money is liable. It's not worth it for the legal industry otherwise. The lawyer earns his pay by convincing the court that whatever established precedent doesn't apply to his case.

[sanex]

Unfortunately.

[cortesoft]

Also because Google is the one with a lot more money than whoever was using Gemini.

[redanddead]

they're very worried about liability, it used to be a small thing, now it's as important as being on the frontier

sad to see, bc China doesn't give a fuck about liability, this is a structural disadvantage

the labs don't feel very protected by government, meanwhile the chinese government is yet again fostering protectionism

american industry keeps getting fucked by dubious lawmakers

[varispeed]

> Things that are illegal are still illegal and we have professionals to deal with crimes.

This is quite naive take though. The direction of travel is more fascism in Western governments where duties of traditional policing are taken over by big corporations whilst police forces are being gutted and made impotent.

[sanex]

My small town police force has an MRAP, definitely not impotent.

[mannanj]

Maybe control is also profitable.

[gordonhart]

> I don't understand why we would turn the models into law enforcement officers

It's a simple corporate risk minimization strategy. Just look at how universally despised Grok is on HN. Not because it's a bad model, but because it has less aggressive alignment which means it can be coaxed into saying things that get Xai pilloried here and elsewhere.

[Wowfunhappy]

I just think Grok is a bad model. I haven't had success with it.

[bilbo0s]

This.

I tried them all.

Grok was worse than even some of the more mediocre open models at actually doing anything. (At least anything tech work related.) GPT and Claude just do what I ask most of the time. With grok, it’s like a chore just getting it to understand the question.

You’re pulling your hair out trying to figure out what on earth you need to do to land in the right place in whatever topsy turvy embedding grok is using?

[noelsusman]

It's mostly just a bad model. Plenty of people would be willing to overlook the baggage if the model was even marginally better than the competition.

[toraway]

I also used to see Grok boosting/slack-cutting on here/Reddit constantly back in Peak Subsidy when xAI was giving out hundreds of dollars of credits for free per month.

After they killed that and then stopped handing out free model access to users of every Cline fork for weeks following model releases, vibe coder hype moved back to Chinese models for cost and the SOTA models for quality.

[kelnos]

Agreed. There's are plenty of instances where people here on HN do mental gymnastics to justify using a truly good product when the company that builds it is morally bankrupt.

Not a criticism (I probably engage in that sort of thinking myself sometimes), just something I've observed. If Grok were actually good, we'd see that phenomenon here, but we don't.

[DANmode]

I just read a bunch of compelling “Grok is better at this” use cases in a thread yesterday.

I’m not rushing towards it, but, had to mention.

[ascorbic]

No, they've clearly put a lot of work into alignment. It's just that they've been trying to align it with Elon Musk rather than Amanda Askell. Unfortunately the more anti-woke they try to make it, the worse it seems to perform.

[skeledrew]

> Unfortunately the more anti-woke they try to make it, the worse it seems to perform.

Probably because being anti-woke generally goes hand in hand with going against facts and logic. Cull the "woke", lose the facts+logic. Not that they care about that anyway.

[lostdog]

Grok is despised because it has more aggressive alignment.

[igravious]

to what does the "it" in "I couldn't get it to turn on a debug flag" refer to?

[sanex]

Composer

[ifwinterco]

Software engineering is one thing but if you look 10-20 years into the future and everyone can run models equivalent to today's SoTA locally with zero monitoring or censorship, that could... not be good.

Some people will use them responsibly but a lot of people will not.

LLMs are already frying some people's brains and there are some human desires that should not be encouraged

[blubber]

That's why there won't be any local models in 10-20 years. The latest Chinese models are already hosted on proprietary clouds.

[regexorcist]

That's a wild assumption and most certainly wrong. Open models will continue to evolve with or without Chinese labs.