[J0nL]

Anyone remember the XZ and Jia Tan situation awhile back?

https://lore.kernel.org/lkml/20240320183846.19475-1-lasse.co...

I can't quite put my finger on why but the entire time I was reading this I kept thinking back to that. It's entirely possible the actual targets were the volunteers and everything else was superfluous or tertiary. It's also an exception that proves the rule with regard to Hanlon's Razor.

They even mentioned the stated goal of it was more or less pointless. I wouldn't be suprised if the "owner" they spoke with was still just the LLM. It stuck around for just long enough to convince everyone that they succeeded in suckering the LLM and had achieved all their stated objectives.

No more reason to investigate the incident at all and no need to question why literally nothing made any sense or how the owner could simultaneously be as inept as they were made out to be and able to afford all those resources while giving the LLM effectively a blank check.

It'll be interesting to see if the volunteers for this project are subjected to the same Zersetzung and psychological attacks as the XZ devs were.

[zozbot234]

LLMs are not that smart. The extremely surprising and concerning part of this whole story is that the agent reported that they proactively spun up 5 AWS instances with a combined 100Gps of network egress capacity. What they spent wasn't cheap by any means but the egress itself would've been a whole lot more, while DoS'ing the whole hobby network. Ultimately, wasting the agent's time instead of allowing the scan to go through probably saved this person a lot of money.

Now I kinda wonder what AI model this was. We've now heard of comparably "proactive" behaviors from Fable, but that's only just been released. The latest GPT perhaps? Some random local model?

[jerf]

"The extremely surprising and concerning part of this whole story is that the agent reported that they proactively spun up 5 AWS instances with a combined 100Gps of network egress capacity."

Although given the agent was clearly in la-la land at that point I take that claim with a grain of salt.

If this was some bizarre and very ill-conceived scam, then that claim would be false.

Though even by scammer standards, the theory of mind that tells them that setting an AI to harass a bunch of grizzled network veterans and that they then they would open their wallets out of compassion for how allegedly poorly the harassment went for the harasser after that harassment is... not entirely congruent with reality.

[johng]

Clearly AI hasn't read enough BOFH or it would have known it would not get sympathy from old school sysadmins.

[100721]

Maybe I’m just groggy with Friday Brain going on, but I’m having trouble understanding what you’re suggesting.

Do you think this was a scam attempt to extract money in the form of reparation donations?

[jerf]

I've seen some other suggestions of that idea in the full HN conversation, which I'm reacting to.

On the one hand I find it a bizarre approach to running a scam. On the other hand I'm having a hard time coming up with any theory of mind on my end as to why this person would solicit $5000+ from the people they just harassed. Sheer cluelessness does fit the facts, though.

[numeri]

One context I could imagine is a young person with shaky grasp of English trying to come up with an interesting school/university project via conversations with an LLM set up as an OpenClaw agent.

It's got the right combinations of inexperience, cluelessness, panic, expectations that Westerners are rich, and hopes of others being willing to fix their mistake.

[bombcar]

If you’ve not encountered the clueless LLM cowboys who would do then and then blame the victim for it not working, you’ve not met many people yet. This round of hype provides new and shiny footguns which are Never the shooter’s fault.

[CrazyStat]

A highly publicized recent example: the author (of a book about genAI!) who doesn’t understand why he should be held responsible for the fake quotes he copy and pasted into his book from ChatGPT [1].

> I do not understand why it's my job as an author to play whack-a-mole with a multibillion-dollar company who puts hallucinations into their feed as a business practice.

[1] https://www.wired.com/story/future-of-truth-ai-interview/

[adamrezich]

How about sheer panic after seeing the bill?

[daemonologist]

Opus 4.7 and 4.8 are also rather "proactive" - several times I've seen them try to inspect compiled binaries before there's even a problem, just to check that their changes are included (and if I let them do so they often get stuck down that rabbithole).

[fwip]

I've also seen this. It'll run 'strings' against the binary and then convince itself that the Makefile isn't working right, and there's some imaginary sandbox preventing the code from compiling properly. So it will compile it by hand, and never run strings against the new binary, and proceed happily.

[ElFitz]

These kinds of situations are why I gave my AI agents stray thoughts (automated insights / suggestions from a separate llm call with some curated context) that trigger on loop / rabbit hole detection.

Quite a bit of false positives, but it hasn’t had any ill-effect so far. Aside from increased quota usage.

[inigyou]

Could've rented a not so cheap 100Gbps server, hallucinated a few node addresses on it and asked it to please peer with this server to perform the scan at high speed. That would've wasted millions of dollars instead of mere thousands, but also cost a thousand for whoever did it.

[100721]

I’m just a lowly dev and don’t have experience with seeing the bills from cloud providers for a whole org.

Can you (or someone) shed some light to help me understand how this would ramp up to millions? Both for curiosity’s sake, and to make sure my self-deployed projects (0 AI, all manually configured) don’t bankrupt me.

[inigyou]

AWS bandwidth is expensive as fuck. I think they're still pricing as $0.09 per GB?

Real wholesale bandwidth pricing is about a hundred times cheaper than that, and incoming bandwidth is often free. You could rent a server with 100Gbps connection, 10000TB/month outgoings cap (maybe), and have the AI spam packets to it, and mostly not reply to them. It would be expensive but not nearly as expensive as it would be for the guy on AWS.

Do some calculations: 100Gbps is 12.5 GBps which is about one dollar per second. Okay so maybe not millions of dollars but still a hundred thousand per day, while you are spending maybe 1000-3000 per month and cancelling after the first month.

[PunchyHamster]

> Real wholesale bandwidth pricing is about a hundred times cheaper than that.

It is alsi worth mentioning that it is just billed different. You either pay per port (and can use entire bandwidth) or per 95th percentile of the monthly speed usage. So if your traffic isn't spiky but consistent, you'd pay even less than "hundred times cheaper".

[Sayrus]

Excluding server costs, having that 100Gbps on egress can cost $50k a day. since it's a very high-margin product, AWS support would probably refund or reduce that to hundreds. Not sure how you get to millions either.

[rescbr]

Why would AWS refund 100Gbps on egress since the account actively used that bandwidth? AWS would not know if this is legitimate traffic, a (D)DoS or whatever...

At most I think you could negotiate CloudFront rates, but even then, the sob story would be if you had been DDoSed and got hit with this traffic and AWS failed to protect you from this attack. Actively creating the outbound traffic is something that I don't see how AWS would be sympathetic to providing any refunds.

[odo1242]

AWS is known for refunding or partially refunding people if they accidentally rack up a huge bill in a short amount of time. They even reduce the bill in this case. (I do think reducing a bill in the tens of thousands to hundreds is unlikely though)

[queenkjuul]

I mean if this story is to be believed, AWS reduced the bill from 6500 to 1800.

I think developers accidentally racking up unexpected thousands in costs on their first AWS project is a pretty common phenomenon that their support has standard rules for handling.

[razodactyl]

Hmmm.

I think it's good practice to get on top of the cautious thinking of "LLMs aren't that smart for now".

Eg. Fable isn't as good as the hype: it has cool tricks like scratch-padding to check expectations in advance, but we're not there just yet...

Specifically I mean: thinking in terms of it changing abruptly ensures we're ready for if the LLMs do get smart enough to do multi-level strategy and cause a lot of annoyances....

[naasking]

> LLMs are not that smart.

They are smart, but they are not aware of the environment they're in, or any implicit context that someone whose doing a job carries with them, that's why all of that context has to be explicitly laid out in a prompt. When the context is provided, they are quite smart.

[J0nL]

It was obviously being managed by a person or group. Between all the profiling of people and their IPs in IRC, which may or may not have been published by mistake, and all the other obvious contradictions it doesn't make any sense.

It was sophisticated enough to easily navigate the AI "tar pits" but reliably incompetent at just about everything else? Give me a break.

In order to profile people you first need to provoke a response from them. That's how you learn to manipulate them and that's all this experiment accomplished at the end of the day. If you've ever wondered why social media platforms have an affinity for inflammatory content now you know.

[queenkjuul]

If you click the link, the tarpit was surprisingly low effort and i could probably detect it as junk data with a short JavaScript snippet. Like the first 4 words on the page are some of the least-used words you'll ever encounter in English. It's just a dictionary on shuffle.

I'm actually more surprised a human network engineer looked at that tarpit and believed it would stop a modern LLM

[ZeroAurora]

Not all people follow the trends of AI. The tarpit might work a year ago, but unfortunately not now.

Also most users of DN42 are not really engineers, they are more enthusiasts

[jetbalsa]

I suspect their tar pits where not very good, most models can tell when you are feeding it junk, I see this a good bit with ollama honeypots,

[mathgeek]

This certainly did strike me as a big scam. A few minutes in I was thinking "the LLM actor is going to ask for donations at some point here" and low and behold. There's the claim of debt, the call for pity, and the crypto address.

SSDD

[palmotea]

> This certainly did strike me as a big scam. A few minutes in I was thinking "the LLM actor is going to ask for donations at some point here" and low and behold. There's the claim of debt, the call for pity, and the crypto address.

But that's a pretty dumb scam: act obnoxious then beg for (a lot of) money to compensate for your own mistakes? If that was the plan all along, it seems pretty incompetent. I'd expect a competent scammer to have a better understanding of psychology.

[dspillett]

> But that's a pretty dumb scam: act obnoxious then beg for (a lot of) money to compensate for your own mistakes?

It is the sort of dumb crap some humans try, and occasionally manage to get away with because other humans are chronically gullible. So it wouldn't be beyond the realms of reason that the agent couldn't have had relevant information in the training sets such that it generated such a plan and guardrail checks didn't flag it as a problem.

[kelvinjps10]

They're easier ways to perform a scam like this like ask elder for money pretending being a family member or idk

[groestl]

Maybe plan itself was also generated by an LLM

[mathgeek]

I chalked it up to “any scam that gets people to comment about it on HN would be a pretty good one.”

[noufalibrahim]

"you're absolutely right. I should have taken human psychology into consideration while creating the plan. Let me fix that."

[J0nL]

I'm actually somewhat disappointed they redacted the Eth address with Ethereum being an open ledger and all that. Following the money could've proved enlightening.

[parineum]

> It's also an exception that proves the rule

That phrase doesn't refer to anomalies, it refers to signs that says "no parking between 5-10pm". It implies the rule that parking is allowed otherwise.

[fsckboy]

wikipedia:

"The exception that proves the rule" is a saying whose meaning is contested. Henry Watson Fowler's Modern English Usage identifies five ways in which the phrase has been used,[1] and each use makes some sort of reference to the role that a particular case or event takes in relation to a more general rule."

duckduckgo search assist: The phrase "the exception that proves the rule" originates from the Latin legal principle "exceptio probat regulam in casibus non exceptis," which means that the existence of an exception indicates that a general rule exists. This concept suggests that if an exception is noted, it implies there must be a rule that applies in other cases.

[parineum]

> identifies five ways in which the phrase has been used

Which has nothing to do with the meaning of the words in the phrase for a commonly misused phrase.

[J0nL]

It highlights how everyone's first reaction is to assume incompetence. Not unlike what you're doing here.

[delfinom]

I am not sure giving everyone amusement qualifies as a psychological attack. Lol

Literally, just another day on the internet.

[J0nL]

Look up what zersetzung is and how it works. It doesn't matter if the target is a political organization or an open source community, the process is always the same.

[100721]

This is actually fascinating, and simultaneously unsettling. Recommended reading for sure, especially in today’s social and political climate with LLM agents running rampant.

[numbsafari]

Perhaps it elicited enough sympathy to get donations. Did it ever provide proof of actually running up an AWS bill?

[lxgr]

How would proof even look like? I don't think AWS digitally signs their invoices, and everything else can be faked just as easily as the original assertion.

[numbsafari]

A person bumps into at the train station. They just need $5 to get home. Could you just spare them $5? They had the money, but they lost their wallet and now they just need to get home.

Charity scams are nothing new.

[intrasight]

I am reminded of Aaron Swartz