[ineedtocall]

The prompt can be useful in crafting attacks that rely on prompt injection. For example- and this doesn't work- an attacker can ask a user to write a specific question to the AI and the answer could contain malicious code(like an XSS).

[swyx]

it's really not that big a deal, and the defenses against it (like you would XSS) is the stuff of regular software engineering anyway (eg sandboxing generated code, authz and rate limiting).

for more on why reverse prompt engineering is overrated: https://news.ycombinator.com/item?id=34165522

[ineedtocall]

I like the cut of your gib.