[ofjcihen]

I was one of the “lucky” few to witness the school of slur-fish.

Being in security I laughed because of how egregious it was but also because I knew someone on HN with some actual time on their hands to help properly would be along soon.

I also appreciate this post mortem. Vibe-coded anything in prod is a lot of my work load in IR these days but it was nice to see such a low stakes project properly documented.

[bombcar]

People will be quick to jump on the "it was vibe coding's fault" but at least two of the issues are pretty common even in designed systems without AI - leaving in a "test admin" access and verifying tokens but not cross-checking them.

[JohnMakin]

This is pretty reductive of the actual problem people typically complain about with vibe coding - It produces very workable prototypes fairly quickly and without a lot of hassle. Great! The problem is, and this is a great example (of many) where someone mistook the working prototype with a system that was ready for production. The JWT thing in particular is not really a mistake many people who work on that kind of thing would make.

People need more understanding of the risks of vibe coding and YOLOing to prod with these tools. They are powerful, but like all powerful tools, can be wielded irresponsibly.

[serf]

it's just incompleteness -- a human issue.

most in-use LLMs prompted with a simple "You're in charge of infrastructure security, let's review possible problem points" would have uncovered this.

I wouldn't fault a compiler for erring when someone left out a period; i'd tell the person to start including it -- but for some reason the expectation for LLMs is hands-off work ; I guess we're just in that phase of the hype at the moment.

[bccdee]

> I wouldn't fault a compiler for erring when someone left out a period

I'd fault it if it silently injected multiple serious vulnerabilities.

[akdor1154]

> for some reason the expectation for LLMs is hands-off work

The expectation is the same as the expectation for self driving: users expect it to be fully hands off, even when they are explicitly told they need to keep their hands on the wheel.

This is because it's tricky, tedious, and unejoyable to thouroughly vet the actions of a machine in realtime.

[floydnoel]

very interesting- i actually enjoy monitoring claude code and telling it when it is going the wrong way on something. i also don’t mind monitoring the car doing its lane keeping, perhaps it is an autism trait?

[bombcar]

Sorry to be the one to tell you, but you might be a born manager ;)

[autoexec]

I think it's pretty reasonable to expect AI to produce systems with issues "pretty common even in designed systems without AI" because that's what AI was trained on.

[JohnMakin]

But that isn’t the expectation or what is being marketed

[edoceo]

I expect these AI and LLM to be, basically, a middle of the bell-curve type producer of code. Just like their other output. Not terrible, not exceptional, just what a Mid could do - only faster.

Not sure what's being marketed, but I expect mediocre.

[brookst]

Being marketed by who? Be specific.

[ofjcihen]

“Be specific.”

Am I the only one that feels like it’s really condescending when people say this on the internet?

It sounds like something you would see on a community college writing assignment

[ungreased0675]

It does sound condescending. I think the sentiment is important though. Asking someone to be specific can help them think clearly. What’s a nicer way to do that?

[degamad]

Dario Amodei: https://www.windowscentral.com/software-apps/work-productivi...

[bombcar]

By Brian Sheltzer, 302 Main St, Chicago, Illinois.

[throwanem]

'By whom.'

[siva7]

I thought checking a token against the cert is actually called verifying or is noawadys verifying just if it looks like a token it maybe a valid token?

[stavros]

Not even a screenshot, I really wanted to see the swastikarp.

[throwanem]

I actually did see one, while the site was #1 and well before the overnight excitement. (Good grief, even at this late age I have something to do with my Friday evenings...)

I don't really know what you want me to tell you about it. The swastika per se as I recall had to be drawn backwards, because there is no meaningful overlap between its outline and that of a fish, so unlike the penis case this is very easy for the classifier. It wasn't clever and it wasn't funny. Several people reported it and it quickly disappeared, whereupon apparently someone decided we shouldn't have nice things, or not for a little while at least.

[stavros]

Yes but what was the incident like, in general? Was it just swastikas, or words too, or what? When I saw it, it was just normal fish.

[ofjcihen]

I got you.

So think of 4 Chan but fish. Like the entire screen.

50 of the fish are just regular fish with slurs written in them.

There’s like 10 swastika fish moseying around.

And then you have the odd fish with like an Isreali flag with…let’s say stereotypical features for the face.

[stavros]

Hahaha that is an excellent description, well done. It's as if I was there.

[throwanem]

Quoting from the article's third paragraph:

> But if you had the displeasure of viewing my website between the hours of 2AM (20 minutes after I went to sleep) and 8AM (when I woke up) EST on Aug 3, then you would have seen chaos. Every single username was transformed to a heinous slur, many unsavory fish had made it into the fishtank, and many beautiful fish were gone.

At some point, asking for more starts to seem like rubbernecking at a car crash, you know?