[throwaway260704]

Using a throwaway account for obvious reasons, but I’m very involved in this space using LLMs from multiple providers. I’m aware of at least two instances in which the intermediate infrastructure “swapped” responses, once impacting Claude models and once impacting GPT models, from two different providers.

One gave us a proper postmortem in which their API gateway was incorrectly handling HTTP 100 status codes, putting them into an error state where there was effectively an off by one error - you would receive the response to the prompt that came in before yours and would pay it forward (your response would go to the next caller).

The other instance never had root cause explained to us, and we were just told to trust it wouldn’t happen again.

Both of these are from $1T+ companies.

ZDR wasn’t compromised in these cases since it was responses being swapped in flight. I wouldn’t be surprised if this is a similar issue - it’s not that data is being retained, it’s just not being safely isolated in intermediate infrastructure.

[pocksuppet]

This attack is called "HTTP desync" or "request smuggling". It's often done intentionally by a client to try and spy on other clients' responses.

Every time you multiplex requests from multiple clients onto one upstream connection, you are probably vulnerable to this, because (despite its superficial simplicity) HTTP is just too complex to reliably match the requests and responses to upstream.

For example a desync can be triggered in some systems by having more than one Content-Length header, by mixing Content-Length with chunked encoding, or by passing an HTTP/2 header called Content-Length that doesn't match the actual content length.

Here's a DEF CON talk (6 years ago) on this topic: https://www.youtube.com/watch?v=w-eJM2Pc0KI

The same attack has been applied to SMTP by messing up the line endings surrounding the end-of-message delimiter, where it's called SMTP smuggling. It may also apply to other protocols.

[markasoftware]

Very true, this was likely an attack. Worth noting that mr kettle has done a defcon talk nearly every year on some variant of this attack, the most recent one titled "HTTP/1.1 must die" because he rightfully believes that switching to the binary headers of http/2 (specifically in reverse proxy connections to upstream servers) is the only way to systematically prevent these.

[albinowax_]

I’ll be back next month with a load of fresh vectors in “Can AI Do Novel Security Research? Meet the HTTP Terminator”

https://portswigger.net/research/talks?talkId=36

Maybe my last presentation on the topic! Possibly.

[bostik]

Or as the Risky Business guys crystallise it: "James Kettle breaks the internet. Again."

[pocksuppet]

Why the reference to AI? This looks like standard security research.

[albinowax_]

If you follow the link, the presentation abstract should hopefully answer that question!

If that doesn’t help I guess you’ll need to wait for the whitepaper to land but I can assure you I didn’t just do my normal research then add AI to the title for clicks :)

[tejusarora]

Woah. Sounds plausible. However, wouldn’t that still be an implicit violation of ZDR since now the response is possibly egressed out of the enterprise network? So if I were working with PHI, the response egress is a potential violation of HIPAA even though claude didn’t retain anything — but the whole Point was to comply with HIPAA. Thoughts?

[nightpool]

I hope you reconsider your use of a throwaway account for this sort of comment—This sort of feedback and experience is very valuable and if the culture of the company you're in or the companies your working with discourage this sort of feedback, it's only going to create chilling effect that prevents more people from coming forward

[throwaway260704]

Would prefer to stick to a throwaway, sorry. It’s not that such feedback is discouraged by my company, it’s been very much escalated. However the response to these incidents had legal repercussions and my replies here aren’t subject to attorney-client privilege.

While whistleblower protections (at least used to) safeguard against government persecution, I work for a private company and don’t want my livelihood risked. I did what I could to escalate through official channels.

[rsync]

Actually, it’s not obvious why you’re using a throwaway account…

Every emergent behavior from these actors - whose claim to positive moral values is barely plausible - should be reported, discussed, dissected and critiqued early and often.

[zymhan]

>should be reported, discussed, dissected and critiqued early and often.

And why does anonymity detract from any of that?

[DANmode]

Your points don’t reference each other.

Yes, the discussion should be had constantly.

But: Should this person potentially have their life messed up because they pointed out the emperor has no clothes?

[throwaway260704]

I like being gainfully employed, and the best position for me to push for genuine AI safety is within an influential company in the space.