[nubinetwork]

I have a hard time believing that Claude instantly figured out this was malware...

I've fed it obfuscated JavaScript before, and it couldn't figure it out... and then there was the time I tried to teach it nftables... whooo boy...

[rgovostes]

If you read the transcript it repeatedly made the incorrect assertion (hallucinated) that it’s totally normal for Claude Code to use Base64 armoring.

It’s not surprising it can “read” Base64 though; such was demonstrated back in GPT-3 days. Nontrivial obfuscation might not be one-shotted, but Claude has access to a code interpreter and can certainly extract and step through the decoder routine itself as a malware analyst would.

nftables is a different problem though. It’s apparent that if something isn’t well understood—i.e, there are tons of badly-formed examples on StackExchange—LLMs will fail to learn it too. I’ve seen this with things as “simple” as Bash string interpolation rules like ${var:+blah}. More often than not I’m humbled when I think I’ll learn it better and then find myself swearing at poorly-written documentation and patently false Q&A advice.

[ares623]

I think the usual response to that is "have you tried again recently?"

[nubinetwork]

3 weeks ago?

[onepunchmob]

They are really good at this, had codex discover similar malware from another supply chain attack months ago because my laptop was running hot. Actually crazy times we live in, I would certainly not be able to discover this without agent help.

[dns_snek]

> and then there was the time I tried to teach it nftables

I think this deserves a short story!

[nubinetwork]

My home router is apparently special because I want a private dmz and inbound blocklists... firewalld couldn't do what I wanted, so I tried nftables, and that went down a rabbit hole that made me wish iptables and ipset weren't being replaced.

[Filligree]

Except in this case the code wasn't obfuscated, right?

[phyzome]

Hard to know with this nondeterministic shit.