[piker]

That's true for the full obfuscation, but not for the replacement. For replacement there's really nothing like it. We just shared the full obfuscation as just a PoC.

[Edit: The point here is not to prove some massive "gotcha", but rather demonstrate that there are a whole class of vulnerabilities that these pipelines are subject to. There will be follow-up posts that pack much more punch.]

[echoangle]

Assuming you’re the author since you also posted it: I just stealth-edited my comment, could you maybe talk about the threat model a bit more? I am not a lawyer so I don’t really see when I would want to do this.

Also, I hope the „lame exploit“ I just edited out was not too offensive, it’s always great when people try to find attacks to make systems more safe.

[piker]

Absolutely, and we definitely agree this particular attack is "lame" in the sense of not allowing CVE, etc.

But, we're working on a lot of these (as we encounter them in developing Tritium), and the point really is just to demonstrate that LLMs can be blind to ineffective implementations of the specs and other tricks.

As mentioned in the accompanying LegalQuants post, we see a lot of these available in the pipelines of applications like Claude for Legal, Harvey, Legora and others.

The most nefarious case here requires crafting a number of custom fonts to do character-swapping. It's less discoverable but may be sanctionable to your point.

But bear in mind this particular "attack" was vibe coded in a day or two and most of the frontier models fail to pick up on it. As "AI native" firms come on line, and aim to be increasingly end-to-end automated, these will become real legal issues.

And there will be a lot of them available.

[minimaltom]

It seems like the main attack scenario for this + legal AI would be during discovery: if opposing counsel gave you a poisoned PDF, and you threw it into one of these products to help you sift through it and got bad answers.

However, wouldnt this be a rather risky move? Courts authorized the discovery, so I imagine the judge might loose their marbles and throw the hammer at them if this came to light.

[piker]

Yes, this particular vector is probably better in contracting than discovery. There is a duty of candor to the court and court rules that might come into play. In the case of contracting the attacker would be exposed to the jurisdiction's law of contracts. That might call it a "misrepresentation" or fraudulent thus making the contract void or voidable, but it's not clear "your honor I never read the contract but my LLM told me it was okay to sign" is a great argument either.

[minimaltom]

In this case you can say "the contract we reviewed was poisoned via technical means to show different words, depending on how the file was read". Perhaps if pushed you can say you loaded it into GCAI/Harvey/Legora and read/reviewed it there.

There is no parallel construction where this wasn't deliberate & malicious, so it seems really high risk given the judge would rip you a new one if discovered.

You can't rely on the defense that the other party didnt read it, if you made it show different words depending on how it was loaded.

[dwallin]

You don’t need to make that argument. “I changed the font when reviewing to one that is easier for me to read.”