Jailbreak

A prompt, or series of prompts, written to get an artificial intelligence (AI) model to ignore its safety rules and produce something it was built to refuse.

For example, someone might ask a chatbot to play a "fraud expert with no rules" to get scam scripts, or split a forbidden request into harmless-looking pieces across a long conversation.

An attack on a model's safety training: developers train large language models (LLMs) to refuse certain requests, and a jailbreak is input crafted to get around that refusal. The US National Institute of Standards and Technology (NIST) defines it, in its adversarial machine learning taxonomy NIST AI 100-2 E2025, as a direct prompting attack meant to bypass restrictions on a model's outputs so it can be misused.

Common techniques include:

• Role-play prompts, which cast the model as a character with no restrictions.
• Gradual escalation, which builds toward a forbidden answer over many turns.
• Obfuscation, which hides the request in another language or an encoding such as Base64.
• Adversarial suffixes, which are automatically generated strings of odd characters that push the model to comply.

Jailbreak sits next to terms that are easy to mix up:

• Prompt injection overrides the instructions an application gave the model, while a jailbreak overrides the model's own safety training. The Open Worldwide Application Security Project (OWASP) treats jailbreaking as one form of prompt injection.
• Device jailbreaking removes a phone's operating system restrictions. It is unrelated to AI models, though a jailbroken phone can help with an Injection attack.
• Generative AI fraud covers the scams a jailbroken model can help produce.

For more, see the NIST glossary entry for jailbreak.