[zozbot234]

LLMs are not that smart. The extremely surprising and concerning part of this whole story is that the agent reported that they proactively spun up 5 AWS instances with a combined 100Gps of network egress capacity. What they spent wasn't cheap by any means but the egress itself would've been a whole lot more, while DoS'ing the whole hobby network. Ultimately, wasting the agent's time instead of allowing the scan to go through probably saved this person a lot of money.

Now I kinda wonder what AI model this was. We've now heard of comparably "proactive" behaviors from Fable, but that's only just been released. The latest GPT perhaps? Some random local model?

[jerf]

"The extremely surprising and concerning part of this whole story is that the agent reported that they proactively spun up 5 AWS instances with a combined 100Gps of network egress capacity."

Although given the agent was clearly in la-la land at that point I take that claim with a grain of salt.

If this was some bizarre and very ill-conceived scam, then that claim would be false.

Though even by scammer standards, the theory of mind that tells them that setting an AI to harass a bunch of grizzled network veterans and that they then they would open their wallets out of compassion for how allegedly poorly the harassment went for the harasser after that harassment is... not entirely congruent with reality.

[johng]

Clearly AI hasn't read enough BOFH or it would have known it would not get sympathy from old school sysadmins.

[100721]

Maybe I’m just groggy with Friday Brain going on, but I’m having trouble understanding what you’re suggesting.

Do you think this was a scam attempt to extract money in the form of reparation donations?

[jerf]

I've seen some other suggestions of that idea in the full HN conversation, which I'm reacting to.

On the one hand I find it a bizarre approach to running a scam. On the other hand I'm having a hard time coming up with any theory of mind on my end as to why this person would solicit $5000+ from the people they just harassed. Sheer cluelessness does fit the facts, though.

[numeri]

One context I could imagine is a young person with shaky grasp of English trying to come up with an interesting school/university project via conversations with an LLM set up as an OpenClaw agent.

It's got the right combinations of inexperience, cluelessness, panic, expectations that Westerners are rich, and hopes of others being willing to fix their mistake.

[bombcar]

If you’ve not encountered the clueless LLM cowboys who would do then and then blame the victim for it not working, you’ve not met many people yet. This round of hype provides new and shiny footguns which are Never the shooter’s fault.

[CrazyStat]

A highly publicized recent example: the author (of a book about genAI!) who doesn’t understand why he should be held responsible for the fake quotes he copy and pasted into his book from ChatGPT [1].

> I do not understand why it's my job as an author to play whack-a-mole with a multibillion-dollar company who puts hallucinations into their feed as a business practice.

[1] https://www.wired.com/story/future-of-truth-ai-interview/

[adamrezich]

How about sheer panic after seeing the bill?

[daemonologist]

Opus 4.7 and 4.8 are also rather "proactive" - several times I've seen them try to inspect compiled binaries before there's even a problem, just to check that their changes are included (and if I let them do so they often get stuck down that rabbithole).

[fwip]

I've also seen this. It'll run 'strings' against the binary and then convince itself that the Makefile isn't working right, and there's some imaginary sandbox preventing the code from compiling properly. So it will compile it by hand, and never run strings against the new binary, and proceed happily.

[ElFitz]

These kinds of situations are why I gave my AI agents stray thoughts (automated insights / suggestions from a separate llm call with some curated context) that trigger on loop / rabbit hole detection.

Quite a bit of false positives, but it hasn’t had any ill-effect so far. Aside from increased quota usage.

[inigyou]

Could've rented a not so cheap 100Gbps server, hallucinated a few node addresses on it and asked it to please peer with this server to perform the scan at high speed. That would've wasted millions of dollars instead of mere thousands, but also cost a thousand for whoever did it.

[100721]

I’m just a lowly dev and don’t have experience with seeing the bills from cloud providers for a whole org.

Can you (or someone) shed some light to help me understand how this would ramp up to millions? Both for curiosity’s sake, and to make sure my self-deployed projects (0 AI, all manually configured) don’t bankrupt me.

[inigyou]

AWS bandwidth is expensive as fuck. I think they're still pricing as $0.09 per GB?

Real wholesale bandwidth pricing is about a hundred times cheaper than that, and incoming bandwidth is often free. You could rent a server with 100Gbps connection, 10000TB/month outgoings cap (maybe), and have the AI spam packets to it, and mostly not reply to them. It would be expensive but not nearly as expensive as it would be for the guy on AWS.

Do some calculations: 100Gbps is 12.5 GBps which is about one dollar per second. Okay so maybe not millions of dollars but still a hundred thousand per day, while you are spending maybe 1000-3000 per month and cancelling after the first month.

[PunchyHamster]

> Real wholesale bandwidth pricing is about a hundred times cheaper than that.

It is alsi worth mentioning that it is just billed different. You either pay per port (and can use entire bandwidth) or per 95th percentile of the monthly speed usage. So if your traffic isn't spiky but consistent, you'd pay even less than "hundred times cheaper".

[Sayrus]

Excluding server costs, having that 100Gbps on egress can cost $50k a day. since it's a very high-margin product, AWS support would probably refund or reduce that to hundreds. Not sure how you get to millions either.

[rescbr]

Why would AWS refund 100Gbps on egress since the account actively used that bandwidth? AWS would not know if this is legitimate traffic, a (D)DoS or whatever...

At most I think you could negotiate CloudFront rates, but even then, the sob story would be if you had been DDoSed and got hit with this traffic and AWS failed to protect you from this attack. Actively creating the outbound traffic is something that I don't see how AWS would be sympathetic to providing any refunds.

[odo1242]

AWS is known for refunding or partially refunding people if they accidentally rack up a huge bill in a short amount of time. They even reduce the bill in this case. (I do think reducing a bill in the tens of thousands to hundreds is unlikely though)

[queenkjuul]

I mean if this story is to be believed, AWS reduced the bill from 6500 to 1800.

I think developers accidentally racking up unexpected thousands in costs on their first AWS project is a pretty common phenomenon that their support has standard rules for handling.

[rescbr]

I do think the discount is believable, but we don't know the line items AWS applied a discount/removed charges.

The developer said the agent deployed multiple CloudFormation templates, I'd bet that AWS waived the charges for the unused resources - like EC2 instances that were idle most of the time, very high margin SKUs, etc.

Now, for 100 Gbps of egress (which didn't actually happen) - and this is grounded speculation - I don't think that AWS would give a discount that is greater than CloudFront rates.

100 Gbps is A LOT of data.

[razodactyl]

Hmmm.

I think it's good practice to get on top of the cautious thinking of "LLMs aren't that smart for now".

Eg. Fable isn't as good as the hype: it has cool tricks like scratch-padding to check expectations in advance, but we're not there just yet...

Specifically I mean: thinking in terms of it changing abruptly ensures we're ready for if the LLMs do get smart enough to do multi-level strategy and cause a lot of annoyances....

[naasking]

> LLMs are not that smart.

They are smart, but they are not aware of the environment they're in, or any implicit context that someone whose doing a job carries with them, that's why all of that context has to be explicitly laid out in a prompt. When the context is provided, they are quite smart.

[J0nL]

It was obviously being managed by a person or group. Between all the profiling of people and their IPs in IRC, which may or may not have been published by mistake, and all the other obvious contradictions it doesn't make any sense.

It was sophisticated enough to easily navigate the AI "tar pits" but reliably incompetent at just about everything else? Give me a break.

In order to profile people you first need to provoke a response from them. That's how you learn to manipulate them and that's all this experiment accomplished at the end of the day. If you've ever wondered why social media platforms have an affinity for inflammatory content now you know.

[queenkjuul]

If you click the link, the tarpit was surprisingly low effort and i could probably detect it as junk data with a short JavaScript snippet. Like the first 4 words on the page are some of the least-used words you'll ever encounter in English. It's just a dictionary on shuffle.

I'm actually more surprised a human network engineer looked at that tarpit and believed it would stop a modern LLM

[ZeroAurora]

Not all people follow the trends of AI. The tarpit might work a year ago, but unfortunately not now.

Also most users of DN42 are not really engineers, they are more enthusiasts

[jetbalsa]

I suspect their tar pits where not very good, most models can tell when you are feeding it junk, I see this a good bit with ollama honeypots,