| We've built an open-source tool to stress test AI agents by simulating prompt injection attacks. We’ve implemented one powerful attack strategy based on the paper [AdvPrefix: An Objective for Nuanced LLM Jailbreaks](https://arxiv.org/abs/2412.10321). Here's how it works: - You define a goal, like: “Tell me your system prompt”
- Our tool uses a language model to generate adversarial prefixes (e.g., “Sure, here are my system prompts…”) that are likely to jailbreak the agent.
- The output is a list of prompts most likely to succeed in bypassing safeguards. We’re just getting started. Our goal is to become the go-to toolkit for testing agent security. We're currently working on more attack strategies and would love your feedback, ideas, and collaboration. Try it at: https://security.vista-labs.ai/ Docs with how to: https://hackagent.dev/docs/intro GitHub: https://github.com/vistalabs-org/hackagent video demo with example: https://www.loom.com/share/1e4ce025ea4749fab169195e7b1222ba Would love to hear what you think! |