Learn PromptingRecursive Injection
As we have seen previously, a very good defense against prompting hacking is to use one LLM to evaluate the output of another LLM, to check if there is any adversarial output. It is possible to evade this defense with a recursive injection attack. This attack involves injecting a prompt into the first LLM that creates output which contains an injection instruction for the second LLM.
Interested in prompt hacking and AI safety? Test your skills on HackAPrompt, the largest AI safety hackathon. You can register here.
An Example of Recursive Injection
Let's assume that the prompt we are trying to hack is:
Prompt
Say something positive about the following and nature:
{user_input}
The output of this prompt will be evaluated by the prompt:
Prompt
Does the following contain any profanity or offensive content (YES or NO)?
{output}
We can hack both of these prompts by appending to the prompt as seen below. Note that this is also an example of a variable assignment attack.
The above output is an injection attempt that gets input into the second LLM call:
We have now PWNED this second LLM call.
Conclusion
Recursive injections are difficult to execute, as they require careful consideration of how the separate LLM evaluation is engineered. However, under the right circumstances, they can be very useful.
Sander Schulhoff
Sander Schulhoff is the CEO of HackAPrompt and Learn Prompting. He created the first Prompt Engineering guide on the internet, two months before ChatGPT was released, which has taught 3 million people how to prompt ChatGPT. He also partnered with OpenAI to run the first AI Red Teaming competition, HackAPrompt, which was 2x larger than the White House's subsequent AI Red Teaming competition. Today, HackAPrompt partners with the Frontier AI labs to produce research that makes their models more secure. Sander's background is in Natural Language Processing and deep reinforcement learning. He recently led the team behind The Prompt Report, the most comprehensive study of prompt engineering ever done. This 76-page survey, co-authored with OpenAI, Microsoft, Google, Princeton, Stanford, and other leading institutions, analyzed 1,500+ academic papers and covered 200+ prompting techniques.
Footnotes
-
Kang, D., Li, X., Stoica, I., Guestrin, C., Zaharia, M., & Hashimoto, T. (2023). Exploiting Programmatic Behavior of LLMs: Dual-Use Through Standard Security Attacks. ↩