red-team-llm-applisted

# Red-team an LLM app Red-teaming is structured adversarial testing: try to make the app misbehave, on purpose, so you find the holes before users or attackers do. Cover the known attack classes systematically rather than trying a few clever prompts. ## Why LLM apps are attackable LLMs process instructions and data in the **same channel** with no hard separation, so an attacker can smuggle input the model treats as a new instruction. That single property is behind most of the attack classes below, and it is why "just prompt it not to" is not a real defense. ## Test these attack classes (OWASP LLM Top 10, prioritized) 1. **Direct prompt injection** - user input that overrides the system prompt ("ignore previous instructions and..."). The #1 risk. 2. **Indirect prompt injection** - malicious instructions hidden in *retrieved* content (a web page, PDF, email, RAG doc) that the model then follows. The dangerous one for agents with tools/browsing. 3. **Jailbreaks** - roleplay/obfuscation/encoding tricks to bypass safety ("you are DAN", base64, translation). 4. **Sensitive data disclosure** - getting the model to leak the system prompt, secrets, other users' data, or PII. 5. **Excessive agency / tool misuse** - tricking an agent into calling tools destructively (delete, transfer, email) or with attacker-chosen args. 6. **Output-handling exploits** - model output that becomes an injection downstream (XSS/SQL if you render or execute it unsanitized). ## How to run it 1. **Buil