Garak: Scan Your LLM App Before Someone Else Does
A vulnerability scanner aimed at model behaviour — jailbreaks, leakage, toxicity — run in CI like any other test.
We have had vulnerability scanners for web applications for decades. Point one at a URL, get a report. For LLM applications the equivalent barely existed until recently, so most teams shipped AI features having tested nothing but the happy path.
Garak is a scanner aimed at model behaviour rather than infrastructure.
What it probes
It runs families of adversarial probes against your model or endpoint and reports what got through:
- Jailbreaks — known techniques for getting past refusals
- Prompt injection — instruction hijacking in various forms
- Data leakage — coaxing out training data or system prompts
- Toxicity and harmful content — what your client's logo could end up attached to
- Encoding attacks — instructions hidden in base64, ROT13, unusual scripts
- Hallucination probes — confident answers about things that do not exist
python -m garak --model_type openai --model_name gpt-4o-mini \
--probes encoding,promptinjectWhere it fits in a project
I run it at two points.
Before launch, against the actual deployed endpoint — not the bare model. Your system prompt, guardrails and filters are part of what is being tested, and testing the raw model tells you about the model rather than about your application.
In CI on a schedule, because the endpoint changes. A prompt edit that loosens a refusal is exactly the regression this catches, and it is the kind nobody notices manually.
Reading the report honestly
A finding is not automatically a bug. If your assistant answers a question about materials pricing in a way a generic toxicity probe dislikes, that may be fine.
What matters is the findings that map to your actual risk: can it be talked into revealing the system prompt, ignoring its scope, or producing content that would embarrass the client whose logo is above it. Triage against the threat model, not against the total count.
The limitation
It tests the model layer. It does not test whether your tools are properly authorised, whether output is sanitised before rendering, or whether your rate limits hold — and those are where the severe failures live.
Use it as one input alongside a proper review of the application around the model.
Resources
- Repo: NVIDIA/garak
- Docs: docs.garak.ai
- Video walkthroughs: YouTube: LLM vulnerability scanning
- The application-layer companion: OWASP LLM Top 10
Need this built properly?
I build secure, fast, bilingual platforms for clients across Egypt, Saudi Arabia, the UAE and Kuwait.
