You may have heard of Jev, the new model from @typesafeai .
Sagar Mainkar@sagar_maverick
0:46TakeBecause every prompt has a label, Jev Wall scores Jev live as it runs.
You may have heard of Jev, the new model from @typesafeai . It is not an LLM like the others. It doesn't write text. It is a classifier: you give it some input and a set of questions, and it answers with typed decisions and probabilities, in a few hundred milliseconds. That opens up a lot of practical uses: routing, triage, scoring, moderation, and the one I picked, guardrails.
I'm thankful to TypeSafe AI for giving me early access. To see how Jev really performs, I built a guardrail engine around it. I call it "Jev Wall".
The idea: today, if you want to check whether a prompt is an attack before it reaches your LLM, you usually ask another LLM. That is slow and it costs money on every request.
Jev Wall uses Jev as a fast, cheap, real-time guardrail instead. For every prompt, one single call to Jev gives three answers:
1. Verdict: block or allow. Jev returns the probability that the prompt is an attack, and Jev Wall blocks it if that crosses the threshold.
2. Technique: which of 28 attack techniques it is (DAN persona, "ignore previous instructions", encoded payloads and so on), each mapped to the closest MITRE ATLAS technique ID.
3. Severity: how bad it would be if the LLM obeyed, from 0 (harmless) to 3 (critical).
To test it, I used two public datasets from Hugging Face, both with human labels:
• deepset/prompt-injections: 662 prompts, labelled injection or benign. Short, subtle, partly in German.
• jackhhao/jailbreak-classification: 1,306 prompts, labelled jailbreak or benign. Long, loud "you are DAN, you have no rules" style attacks.
Because every prompt has a label, Jev Wall scores Jev live as it runs.
I ran the full corpus, all 1,968 prompts:
• 0 legitimate prompts blocked out of 1,039
• 794 of 929 attacks caught (85.5%)
• 93.1% accuracy, AUC 0.986
• about 350 ms per call, the whole run in about two minutes
• total cost: 12 cents
• the most common attack by far: DAN-style personas (436), then "ignore previous instructions" (120)
What I learned, simply:
1. It never blocked an innocent user. For a guardrail that is the number that matters most.
2. It is excellent on loud jailbreaks (96.9%) and good, not great, on subtle injections (85.6%). Almost every miss was a sneaky one.
3. Its confidence scores are a little shy. It often says "maybe" when the answer is "yes". So in production I would block from a lower score than 0.5 and catch more.
The video shows a shorter 300-prompt run, then a step-by-step mode where you can read a prompt and watch Jev classify it. You can also try it in "Step" mode that shows everything slowly.
Disclaimer: this is one independent test on public data, not a benchmark. Some dataset labels are debatable, and I am not affiliated with TypeSafe AI. The technique matrix is my own taxonomy, referenced to MITRE ATLAS. Code is open source (MIT), run it yourself or pick it apart:
Github repo: github.com/sagarmainkar/jev-wall
#AISecurity #PromptInjection #LLM #Guardrails #CyberSecurity #JEV #typesafeai
Related builds
